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Overview 


In 2004, Lumina Foundation launched Achieving the Dream, a national initiative aimed at cul- 
tivating systemic change in community colleges, with the ultimate goal of increasing student 
success, particularly among low-income students and students of color. Now encompassing 
nearly 200 institutions across the country, Achieving the Dream aims to help community col- 
leges build a “culture of evidence” by increasing their capacity to collect and use data in order 
to identify barriers to success and to subsequently develop intervention strategies. 

In 201 1 , MDRC and the Community College Research Center published Turning the Tide: Five 
Years of Achieving the Dream in Community Colleges. That report described the implementa- 
tion of the initiative and student outcome trends, through 2009, for the first 26 colleges that 
joined Achieving the Dream in 2004 (called the “Round 1” colleges). This report builds on 
Turning the Tide in two ways. First, it extends the analyses of student outcomes through 2011, 
to include students who were entering the Round 1 institutions during the latter period of the 
colleges’ five-year implementation grants, when institutions were expected to have more fully 
implemented many of their Achieving the Dream initiatives. Second, the report explores varia- 
tion in student outcome trends in Round 1 colleges and reanalyzes the implementation data, 
seeking lessons to inform other colleges that are undertaking refonns. 

Overall, this report finds that average institution-wide student outcome trends remained relative- 
ly stable during the period of study, including during the prolonged recession that began in the 
United States in late 2007. Three colleges, however, stood out for gains on multiple indicators 
of student success. Although this report cannot directly link practices at these colleges to gains 
in student outcomes, their experiences do suggest lessons for community college practitioners, 
in addition to new directions for research. In particular: 

• Each college focused on specific student subgroups, and each coordinated multiple refonn 
efforts around their chosen subgroup. 

• In later years, after gaining experience with the initial subgroups, each college expanded the 
reach of new practices to include larger groups of students or faculty. This focus was sup- 
ported by targeted professional development for faculty and staff involved in the work. 

• One college also used its reaccreditation process to help coordinate its refonn efforts toward 
achieving a common set of goals. 

This report marks the end of a long-tenn study on the first colleges to join Achieving the 
Dream. The initiative was designed to tackle persistently low rates of student success in com- 
munity colleges. The Round 1 colleges have demonstrated that even while change can occur at 
the institutional level, for example, becoming a more data-driven college, substantially improv- 
ing institution-wide student outcomes is more challenging and much harder than was envisioned 
at the start of the initiative. 
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Preface 


Traditionally focused on increasing access to higher education in the United States, community 
colleges currently serve millions of low-income, minority, and nontraditional students, yet 
student success remains elusive. Almost half of all first-time community college students are 
not enrolled in any institution and have not received a degree or certificate six years after first 
entering college. Moreover, until relatively recently, most community colleges had neither the 
capacity nor the incentives to collect data on student perfonnance and to analyze it to inform 
college practices and policies. Ten years ago, Lumina Foundation and a group of partner 
organizations — the American Association of Community Colleges; the Community College 
Leadership Program at the University of Texas; the Community College Research Center 
(CCRC) at Columbia University; Jobs for the Future; MDC, Inc.; MDRC; and Public Agenda 
— set an ambitious agenda for change. 

In 2004, Lumina and the partner organizations launched Achieving the Dream, a na- 
tional initiative designed to help community colleges collect and analyze student performance 
data and apply the results to help students succeed. The initiative focused on building a “culture 
of evidence” — one in which colleges routinely use evidence to help their students succeed 
academically. An earlier report published by MDRC and CCRC found that most of the “Round 
1” colleges — the first 26 colleges to join the initiative — made important progress in establish- 
ing a culture of evidence on their campuses. However, it also found that institution- wide student 
outcome trends generally remained unchanged. 

This report follows up on the earlier report and adds to the analysis an additional two 
years of data on student outcomes. As in the previous report, trends in student outcomes did not 
change appreciably. The report also finds, however, that some of the Round 1 colleges made 
gains on multiple indicators of student success, and the report describes those colleges’ stories 
in greater depth to suggest lessons for the field to explore. As a whole, Round 1 colleges have 
demonstrated not only that change in community colleges can occur at the institutional level, 
but also that the challenges to improving institution- wide student outcomes remain substantial. 

Achieving the Dream has evolved into an independent organization that now includes 
nearly 200 colleges, 15 state policy teams, and more than 100 coaches and advisers who 
participate across 32 states and in Washington, DC. There is much work to be done to improve 
the prospects of students entering community colleges, and Achieving the Dream has estab- 
lished a solid data-driven foundation in a strong network of colleges to embark on that work, 
becoming an engine of innovation and experimentation. 

Gordon L. Berlin 

President, MDRC 
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Executive Summary 


Each year, millions of Americans enroll in community colleges, seeking to develop the skills 
necessary to pursue a career or to transfer to a four-year institution. Community colleges serve 
large proportions of nontraditional, low-income, and minority students, 1 and they are designed 
to provide access to a postsecondary education at a low or relatively affordable cost. Yet for 
most students who enter these institutions, academic success remains elusive. Six years after 
entering community college, almost half of first-time students are not enrolled at any institution 
and have not received a degree or certificate. 2 In response to these low rates of student success, 
Lumina Foundation identified community college as a “high-need area” that is ripe for systemic 
refonn and direct assistance. 3 

In 2004, Lumina Foundation and a group of partner organizations — the American As- 
sociation of Community Colleges; the Community College Leadership Program at the Universi- 
ty of Texas; the Community College Research Center (CCRC) at Columbia University; Jobs for 
the Future; MDC, Inc.; MDRC; and Public Agenda — launched Achieving the Dream, a bold, 
multiyear national initiative aimed at improving student outcomes in community colleges, par- 
ticularly among low-income students and students of color. The partner organizations were se- 
lected to help Lumina design and operate Achieving the Dream, which set out to foster funda- 
mental changes in the culture and operations of community colleges. Lumina and the founding 
partners sought to spur a process of institutional change through monetary and professional 
supports from the initiative, combined with colleges’ own investments. This process centered 
on building a “culture of evidence” — one in which colleges routinely use evidence to help then- 
students succeed academically. The partners theorized that undertaking broad-based institution- 
al efforts would ultimately lead to improvements in student outcomes. 

Twenty-six colleges (called the “Round 1” colleges) were the first to join the initiative 
in 2004. In 201 1, MDRC, in partnership with CCRC, published Turning the Tide: Five Years of 
Achieving the Dream in Community Colleges . 4 That report described the implementation of the 
initiative and trends in student outcomes among these 26 colleges from 2004 through 2009. 

Stephen Provasnik and Michael Planty, Community Colleges: Special Supplement to the Condition of 
Education 2008, NCES 2008-033 (Washington, DC: U.S. Department of Education, Institute of Education 
Sciences, National Center for Education Statistics, 2008). 

2 A. W. Radford, L. Berkner, S. C. Wheeless, and B. Shepherd, Persistence and Attainment of 2003-04 
Beginning Postsecondaiy Students: After 6 Years (Washington, DC: U.S. Department of Education, National 
Center for Education Statistics, 2010). 

3 Lumina Foundation, “Community College Initiative Overview,” unpublished paper (Indianapolis: Lumi- 
na Foundation, 2002). 

4 Elizabeth Zachry Rutschow, Lashawn Richburg-Hayes, Thomas Brock, Genevieve Orr, Oscar Cerna, 
Dan Cullinan, Monica Reid Kerrigan, Davis Jenkins, Susan Gooden, and Kasey Martin, Turning the Tide: Five 
Years of Achieving the Dream in Community Colleges (New York: MDRC, 2011). 
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This report, the final publication from MDRC and CCRC on the Round 1 colleges, builds on 
Turning the Tide in two ways. First, it extends the analyses of institution-wide outcomes to stu- 
dents who were entering the Round 1 institutions during the latter period of the colleges’ five- 
year implementation grants, when institutions were expected to have more fully implemented 
many of their Achieving the Dream initiatives. Second, the report explores variation in student 
outcome trends at Round 1 colleges and reanalyzes the implementation data in order to inform 
other institutions that are undertaking reforms. This report is a retrospective study of Achieving 
the Dream as it was implemented between 2004 and 2009 at the first 26 colleges to participate, 
rather than an assessment of the initiative’s direct impact on its student outcomes or current ac- 
tivities and programs. Indeed, the initiative now includes nearly 200 participating colleges. 

Overall, this report finds that average institution-wide trends in student outcomes re- 
mained relatively stable during the period of study, including during the prolonged recession that 
began in the United States in late 2007. Three colleges, however, stood out for gains on multiple 
indicators of student success. The practices of these institutions suggest possible lessons for 
community college practitioners, in addition to new directions for research. In particular: 

• Each college focused on specific student subgroups, and each coordinated 
multiple reform efforts around their chosen subgroup. 

• In later years, after gaining experience with the initial subgroups, each col- 
lege expanded its new practices in order to reach larger groups of students 
and faculty. Targeted professional development for faculty and staff involved 
in the work supported this focus. 

• One college used its reaccreditation process to help coordinate its reform ef- 
forts and to work toward establishing a common set of goals. 

Although this report cannot directly link such practices to the three colleges’ gains in 
student outcomes, the practices themselves are especially relevant in the context of new reforms 
currently being undertaken across the field. Future research can be integrated into similar re- 
forms to investigate their impacts. This report concludes by discussing the lessons gleaned from 
the experiences of Round 1 colleges with Achieving the Dream. 


Community Colleges and the Achieving the Dream Model 

Traditionally focused on increasing access to postsecondary education, particularly for low- 
income students, most community colleges did not initially have the capacity or the incentives 
to gather information about their students’ performance over time or about the possible barriers 
to their success. 
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At the time Achieving the Dream was launched, most community colleges across the 
country gathered large amounts of data on students through enrollment forms, placement tests, 
and academic transcripts. Because community colleges typically derive their funding based on 
enrollment numbers, they were adept at compiling information on the number and characteris- 
tics of students who were enrolled in the institutions overall as well as in specific academic 
programs. Community colleges also documented the number of certificates and degrees con- 
ferred each year. It was much less common, however, for these colleges to track students over 
time to determine whether they stayed enrolled and made steady progress toward attaining a 
degree. In addition, these institutions generally did not conduct more detailed analyses to dis- 
cover whether some groups of students experience more difficulty than others in completing 
courses or finishing degrees. 5 

In this context, the vision of Lumina and of the Achieving the Dream founding part- 
ners represented a significant challenge to community colleges. First, as part of the initiative, 
participating colleges were asked to use data in a different way than they normally did. This 
involved examining data more closely to learn whether students are staying in school and 
meeting other critical benchmarks, and to break down these data to determine whether there 
are “achievement gaps” among some segments of the population, such as students of color and 
low-income students. Second, Achieving the Dream aimed to foster fundamental changes in 
community college culture and operations, with the ultimate aim of increasing the academic 
success of these students. 

From its inception, Achieving the Dream emphasized efforts intended to help increase 
the capacity of community colleges to collect and use data to guide institutional improvements. 
The vision of the initiative was to encourage colleges that were relatively inexperienced in using 
data to engage in systematic, data-driven — or data-infonned — decision making. As part of 
this vision, broad groups of faculty, staff, and other employees of the colleges would be in- 
volved in identifying needs, in implementing interventions, and in continuously improving prac- 
tices based on evaluation results. The expectation was that this would give colleges a better idea 
of students’ challenges, thereby enabling them to enact programs and interventions that might 
improve students’ perfonnance. Five key outcomes were targeted: (1) progression through de- 
velopmental education, (2) completion of gatekeeper (introductory college) courses in math and 
English, (3) completion of courses with a grade of “C” or better, (4) persistence, and (5) attain- 
ment of credentials. Achieving the Dream also emphasized this process as a means to overcome 
achievement gaps, particularly among students of color and low-income students. 


5 Thomas Brock, Davis Jenkins, Todd Ellwein, Jennifer Miller, Susan Gooden, Kasey Martin, Casey 
MacGregor, and Michael Pih, Building a Culture of Evidence for Community > College Success: Early Progress 
in the Achieving the Dream Initiative (New York: MDRC, 2007); Vanessa S. Morest and Davis Jenkins, Insti- 
tutional Research and the Culture of Evidence at Community > Colleges, Culture of Evidence Series, Report 1 
(New York: Community College Research Center, 2007). 
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As Figure ES.l shows, the initiative outlined the following steps to guide participating 
institutions in this process of change: 

• First, leaders commit to both making policy changes and allocating re- 
sources to support efforts aimed at increasing student success. 

• Second, colleges use data to understand how students are performing, to 
identify groups that may need extra support, and to prioritize actions. 

• Third, colleges engage faculty, staff, and other stakeholders in using data 
and research to develop intervention strategies designed to address problems 
that the colleges identify as priorities. 

• Fourth, colleges implement and assess strategies by using evaluation re- 
sults to make decisions about whether to expand or refine the strategies. 

• Finally, colleges establish an infrastructure to support continuous, sys- 
temic improvement by institutionalizing effective policies and practices. 
Program review, planning, and budgeting are driven by evidence of what 
works best for students. 

Thus, the colleges were asked to make significant changes in a relatively short time and, 
considering the overall operating budgets that were available, with relatively limited additional 
funds. The five steps outlined above were expected to build on and reinforce one another, such 
that participating colleges needed to master the earlier steps in order to complete the later steps 
and to ultimately engender large-scale institutional change. Moreover, this process was envi- 
sioned as iterative. For example, after bringing to full scale an intervention strategy found to be 
successful, a college would repeat the initial steps of the process to identify and to address new 
problem areas. 


Building on Turning the Tide 

The Tuniing the Tide report found that most Round 1 colleges made important strides in build- 
ing a culture of evidence — progress that most colleges attributed to Achieving the Dream as 
well as to such forces as accreditation processes and other grants. 6 While colleges varied in their 
implementation of the Achieving the Dream model, many enhanced their leadership commit- 
ment to student success, increased their research capacity, and implemented a wide range of 
strategies aimed at improving student achievement, as indicated below: 


bindings summarized in this paragraph and in the next are from Rutschow et al. (2011). 
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Achieving the Dream: Community Colleges Count 


Figure 

Achieving the Dream’s Five-Step Process for Improving Student Success 
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STEPS 


STEPS 

Implement, 


Engage 

evaluate. 


stakeholders. 

improve. 





Improved student outcomes and reduced 

achievement gaps among subgroups of 

students based on five indicators: 

■ Completion of developmental courses and 
progression to credit-bearing courses 

■ Completion of gatekeeper courses, particularly 
first college-level or degree-credit courses in 
math and English 

■ Completion of attempted courses with 
a C or better 

■ Persistence from term to term and year to year 

■ Attainment of credentials 


SOURCE: Field Guide for Improving Student Success (2009), p.l 1, from the Achieving the Dream: Community 
Colleges Count initiative. Copyright 2009 by Lumina Foundation Reprinted with permission. Available 
online atwww.achievingthedream.org/docs/Field_Guide_for_Improving_Student_Success.pdf. 
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• Altogether, 21 of the 26 Round 1 colleges (81 percent) improved their culture 
of evidence over the course of their five-year participation in Achieving the 
Dream. 

• By the end of this period, 1 1 Round 1 colleges (42 percent) had implemented 
most of the practices associated with a strong culture of evidence. These col- 
leges featured broad-based involvement of college administrators, faculty, 
and staff; strong institutional research departments that produced readily un- 
derstandable reports on student achievement; regular evaluations of interven- 
tions to improve student success; and attention to scaling up program strate- 
gies that enhanced student success. 

• Ten colleges (38 percent) had instituted many aspects of the improvement 
process suggested by Achieving the Dream, though not to the same degree as 
those described above. 

• Five colleges (19 percent) were still struggling to implement a number of the 
practices recommended by Achieving the Dream by spring 2009. Weak insti- 
tutional research departments were the primary obstacle to the ability of the 
schools to institute a broad data-driven culture. 

In summary, Turning the Tide documented the important institutional and cultural 
changes that took place at Round 1 colleges during the Achieving the Dream grant period. The 
research also revealed a number of areas for improvement, including the need for an increased 
focus on faculty and staff engagement, particularly among adjunct or part-time faculty; more 
clearly developed guidelines regarding the initiative’s efforts to reduce achievement gaps by 
race and by income level; and a more systematic effort to help colleges expand and institutional- 
ize their efforts to encourage student success. In particular, a key task remained at the end of the 
five-year period: bringing these system-level reforms to the individual level, such that they 
meaningfully changed the experience of large numbers of students. 

This report is the final publication from MDRC and CCRC on Achieving the Dream 
Round 1 colleges. It builds on Turning the Tide by extending student outcome trends to include 
those who were entering these institutions during the laher period of the colleges’ five-year im- 
plementation grants, when they were expected to have more fully implemented many of the 
practices outlined by Achieving the Dream. In addition, this report describes reanalysis of the 
implementation data and of student outcome data to assess whether variations in student out- 
come trends were associated with colleges’ differing pahems of implementation. 

As in Turning the Tide , the analyses of student outcomes in this report are not causal: 
they do not evaluate the impact of Achieving the Dream. Instead, they describe trends in key 


ES-6 



indicators of student success across diverse institutions for eight cohorts of students. Three of 
these cohorts entered college before the implementation of the initiative, from fall 2002 through 
fall 2004, and five began college after the implementation of the initiative, from fall 2005 
through fall 2009. In addition to the colleges’ implementation of Achieving the Dream, several 
important changes occurred during this time period that might have affected students and exert- 
ed pressures on community colleges, making it difficult to isolate the effect of the initiative. No- 
tably, the Great Recession started toward the end of 2007 and, during the period of study, col- 
leges experienced large population shifts. For example, the average full-time enrollment at 
Round 1 colleges increased from just over 6,200 students in fall 2002 to over 8,500 students in 
fall 2009 — a gain of over 37 percent — with the biggest increases occurring in 2008 and in 
2009. Rising enrollments were accompanied by shifting demographics, and the proportion of 
minority students gradually increased. In addition, state funding for higher education was cut 
during this period, driving lower per-student revenue. 7 Finally, nine of the twenty-six Round 1 
colleges did not provide data for the full period of the study, so the infonnation provided here 
does not fully reflect student outcomes for all Round 1 colleges for all cohorts. 8 The colleges 
that did not submit all of the data tended to have lower student outcomes in the earlier cohorts, 
for which they did submit data. 

Achieving the Dream started out with big ambitions. The initiative encouraged colleges 
to develop new interventions, to examine their effectiveness, and to institutionalize those that 
were most promising. While Achieving the Dream emphasizes the importance of disaggregat- 
ing data to inform this process, the partners involved believed that the process as a whole would 
lead to stronger interventions that would together positively affect college-wide student out- 
comes. They also believed that the cultural and institutional changes themselves would engen- 
der changes in student outcomes. Consequently, the partner organizations collectively decided 
that broad, institution-wide outcomes would best reflect these outcomes and that these should be 
used for the evaluation. 

Scaling, however, proved particularly difficult within the five-year grant period. The 
analyses in Turning the Tide showed that relatively small proportions of students experienced 
intervention reforms undertaken by Achieving the Dream during this time frame. Despite the 
notable efforts of colleges to scale up their programs and services, the majority of strategies that 
were carried out remained small in scale, reaching less than 10 percent of their intended target 
populations. This was particularly true when the strategies involved the kind of intensive con- 
tact that might be expected to meaningfully influence students’ perfonnance. The benefits of 
promising interventions, therefore, were frequently extended to only a fraction of students. Con- 

7 Rita J. Kirshstein and Steven Hurlburt, Revenues: Where Does the Money Come From? A Delta Data 
Update, 2000-2010 (Washington, DC: American Institutes for Research, 2012). 

8 Some multi-campus districts, such as Alamo Community College District, are counted as one col- 
lege. 
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sequently, institution-wide trends may not reflect the impact of these reforms on the smaller 
group of students who experienced them. For this report and for Turning the Tide, however, the 
partners’ early decisions about using institution- wide student outcome measures meant that ad- 
ditional data on individual students were not collected. This made it impossible to identify and 
to analyze outcomes for subgroups of students, such as those who participated in Achieving the 
Dream interventions. 

Perhaps unsurprisingly, then, this report shows that with an additional two years of data, 
average college-wide trends in student outcomes at Round 1 colleges remained relatively un- 
changed from pre- to post-initiative cohorts. One exception was that there were modest im- 
provements over time in the rates for completion of gatekeeper English, but other outcomes re- 
mained substantially the same. As with the colleges’ implementation experiences, however, 
there was considerable variation in student outcome trends across Round 1 colleges: some col- 
leges improved on certain outcomes over the five years, and some declined. 

The variation in student outcome trends was also explored through deeper analysis of 
field visit implementation data and survey data. This information was used to investigate the 
relationship between changes in student outcomes and a variety of institutional factors that 
might affect these outcomes, including core activities related to Achieving the Dream. These 
factors included colleges’ leadership strength, aligmnent between institutional goals and re- 
forms, institutional data and research capacity, and professional development training. Although 
changes in student outcomes were not found to be systematically related to such factors, the 
analysis identified three colleges that stood out for their gains on multiple indicators. 

Figure ES.2 shows the distribution of all of the colleges’ gains or declines on each of six 
outcomes. These were (1) completion of developmental English within two years, (2) completion 
of developmental math within two years, (3) completion of gatekeeper English within two years, 
(4) completion of gatekeeper math within two years, (5) persistence from fall-to-fall semesters, 
and (6) attainment of college credentials within four years. Each point in Figure ES.2 represents 
the average change for an individual college for each outcome. The average change is calculated 
by subtracting the average for the outcome for cohorts who entered the college in the pre- 
initiative period (2002-2004) from the average for cohorts who entered the college after the start 
of implementation (2005-2009). Figure ES.2 also identifies where the three colleges that stood 
out — labeled A, B, and C — ranked for each outcome. The gains of these colleges are concen- 
trated in the gatekeeper courses, in fall-to-fall persistence, and in four-year completion. 

The three colleges represent a diverse set of schools with differing student contexts, im- 
plementation strategies, and experiences with Achieving the Dream. They range in size from 
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Pre-post percentage point change 


Achieving the Dream: Community Colleges Count 
Figure ES.2 

Pre-Post Percentage Point Change for Round 1 Colleges, 
by Outcome, 2002-2004 Versus 2005-2009 



Developmental Developmental Gatekeeper Gatekeeper math Fall-fall 4- Year completion 
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Figure ES.2 (continued) 


SOURCE: MDRC calculations using the Achieving the Dream database maintained by JBL Associates. 

NOTES: Calculations for these figures used institutional means for all available data for sample members in the fall 2002 through fall 
2009 cohorts at Achieving the Dream Round 1 colleges. Percentage point change is calculated by subtracting the preintervention (2002 to 
2004) average from the postintervention (2005 to 2009) average. For the developmental English, developmental math, gatekeeper 
English, gatekeeper math, and fall-to-fall persistence measures, 1 site was excluded from cohort 2002, 2 sites from cohorts 2007 and 
later, 4 sites from cohorts 2008 and later, and 12 sites from cohort 2009 because of insufficient data for 2-year outcomes. For the 4-year 
completion measure, 1 site was excluded from 2002, 2 sites from cohorts 2005 and later, 5 sites from cohorts 2006 and later, and 12 sites 
from cohort 2007 because of insufficient data for the 4-year outcome. 



small rural colleges to a large multicampus institution located in a metropolitan area. The insti- 
tutions are located in three different states and have distinctive student populations, ranging 
from a school that serves a largely Hispanic population to a relatively racially and ethnically 
diverse college. 


Lessons for the Field 

Implementation data for each of the three colleges were analyzed in an effort to discern com- 
mon patterns that might be associated with the gains the schools realized. While the colleges 
shared some similar implementation features and challenges, their experiences during the grant 
period were more different than similar. Although no clear patterns emerged, the diversity of 
these experiences suggests lessons for other community colleges that are undertaking refonns. 

Aligning Reform Efforts for Systematic Change 

It may not be enough for colleges to implement a variety of programs and services if 
they are not well aligned. Research in kindergarten through twelfth grade (K-12) education em- 
phasizes the importance of implementing instructional programs in a coherent, cohesive, and 
aligned way, so that reforms work toward the same goals and interact with one another to be 
mutually reinforcing. 9 

Surrounding targeted students with the necessary supports may require such coordina- 
tion. Moreover, the integration of multiple initiatives, policies, and practices may require a gen- 
eral redesign of programs and support services. The literature on organizational effectiveness 
and improvement suggests that no one practice, or even set of practices, may be sufficient to 
improve the performance of an operation as complicated as educating students in a wide variety 
of program fields and types. 10 Improving perfomiance may therefore require fundamental sys- 
temic reform — reviewing and redesigning practices, policies, and processes over time and 
realigning them with new organizational goals. 11 


9 Fred M. Newmann, BetsAnn Smith, Elaine Allensworth, and Anthony S. Bryk, “Instructional Program 
Coherence: What It Is and Why It Should Guide School Improvement,” Educational Evaluation and Policy 
Analysis 23 (2001): 297-321; Geoffrey D. Boreman, Gina M. Hewes, Laura T. Overman, and Shelly Brown, 
“Comprehensive School Reform and Achievement: A Meta-Analysis,” Review of Educational Research 73, 2 
(2003); Anthony S. Bryk, Penny Bender Sebring, Elaine Allensworth, Stuart Luppescu, and John Q. Easton, 
Organizing Schools for Improvement: Lessons from Chicago (Chicago: The University of Chicago Press, 
2010 ). 

10 Bryk et al. (2010); Davis Jenkins, “Redesigning Community Colleges for Completion: Lessons from 
Research on Eligh-Performance Organizations,” CCRC Working Paper No. 21 (New York: Community Col- 
lege Research Center, 2011); Adrianna Kezar, “What is the Best Way to Achieve Broader Research of Im- 
proved Practices in Eligher Education?,” Innovative Higher Education 36, 4 (2011): 235-247. 

"Jenkins (2011). 
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There were some instances in which Achieving the Dream colleges brought about more 
systemic changes to programs and services. However, most of the initiative’s efforts focused on 
the early stages of students’ college experiences and did not address challenges students might 
face in later courses or in staying on track toward graduation. For example, a number of colleg- 
es redesigned their student intake process (for instance, orientation, placement testing, initial 
advising) or enhanced instruction in developmental education or in gateway college-level math 
and English courses (for example, supplemental instruction, learning communities). However, 
few schools focused on curricular and instructional reforms in college-level courses beyond 
math and English, though some did so under the aegis of other initiatives such as accreditation 
or other grant programs. 

A number of Round 1 schools found it difficult to engage large groups of faculty in the 
reform agenda. This limited engagement may be due partly to colleges’ emphasis on the early 
stages of students’ experiences, which may be relevant to only a small proportion of faculty, 
such as those teaching developmental education courses or introductory math or English cours- 
es. Similarly, the broad institutional performance measures used in Achieving the Dream, such 
as persistence and completion, may not resonate with faculty who focus on issues related to cur- 
riculum and instruction in their own disciplines. 12 

Building on Accreditation Processes 

Some of the colleges encountered difficulty in their attempts to engage faculty. One col- 
lege successfully used the accreditation process to engage faculty by focusing on student suc- 
cess measures and by using data to inform improvements in classroom practice. This college’s 
model, which also emphasized the measurement of learning outcomes, was eventually used by 
other departments as well as by the initial group of faculty. Learning more about how colleges 
can integrate a focus on student learning as well as on student progression and completion may 
help to identify ways of engaging faculty as central actors in future reform efforts. This ap- 
proach may also illuminate strategies to incorporate research and data, as well as to implement 
interventions that are data-infoimed, more fully into the reform process. 

Engaging Leaders Throughout Community Colleges 

This report also highlights colleges’ efforts to broaden their leadership framework to in- 
clude faculty leaders and middle managers in the reform process. Effective leadership can occur 
at multiple levels within a college. Presidents and high-level administrators play a key role in 
institutional reform. However, given the decentralized organization of many colleges, the in- 
volvement of deans and department chairs can also be critically important for mobilizing faculty 


12 Jenkins (2011); Rutschow et al. (2011). 
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and staff . 13 Cultivating leaders among the faculty and student services staff may also be an es- 
sential step toward institutional change; such leaders can sometimes spur change more effec- 
tively among their peers and within their departments. Colleges that cultivate strong leaders 
throughout their institutions may have a greater chance of long-term success, even when some 
leaders move on and must be replaced by others . 14 

Starting with Key Student Subgroups 

While improvements in institution-wide student outcomes are likely to require changes 
that affect a large proportion of students, different students may be better served by different 
reform efforts. The three colleges that are analyzed in greater detail in this report demonstrated 
varying approaches to reform, and they often tailored new practices to the specific needs of tar- 
geted students. They also focused multiple strategies on high-need subgroups of students — 
primarily those placed or enrolled in developmental courses. Consequently, these colleges 
reached smaller groups of students and provided them with diverse and coordinated services 
and practices, such as academic assistance provided through advising or through the use of sup- 
plemental supports in the classroom. 

Integrating Research with Reform 

In general, there is a continued need for research on community colleges that is focused 
on reform strategies. Developing future refonns in tandem with rigorous research designs could 
help fill this need. Although institution-wide student performance is unlikely to change substan- 
tially if large numbers of students do not experience something different, even large-scale re- 
fonns will not produce institution-wide change if new innovations are ineffective. Further, there 
is a need to prioritize ongoing, careful, and rigorous evaluations to help college leaders, faculty, 
and staff determine which refonns are effective. Knowledge about program effectiveness and 
common drivers for organizational change is growing, but there is still much to leam — espe- 
cially with respect to institutional change in organizations as complex as community colleges. 
Rigorous research is needed on the causal aspects of specific interventions that directly affect 
the experiences of students. The results of this research (that is, reliable estimates of student 
outcome gains resulting from the new practices) can inform the field in general and individual 
colleges in particular. Such studies can provide colleges with specific and accurate feedback 
that they could use to inform decisions about programs and services. Sharing this knowledge, in 
turn, could help other institutions implement and scale up more effective practices. When re- 
forms primarily target institutional change and reach students only secondarily — such as re- 
fonns that promote increased institutional research capacity and data use — changes in stu- 

13 Kezar (2011); W. Norton Grubb, Basic Skills Education in Community’ Colleges: Inside and Outside of 
Classrooms (New York: Routledge, 2013). 

14 Jenkins (2011). 
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dents’ experiences may occur less directly. In such cases, research on both the process itself and 
on measures of institutional change may be more helpful. 


Conclusion 

Achieving the Dream was founded on a belief in evidence-based decision making that resonated 
with colleges across the country and that persists today. The initiative has helped community 
colleges establish a solid foundation for change. Moreover, it has helped colleges to enhance 
their institutional research and data collection capacities, to develop stronger cultures of evi- 
dence, and to pursue a variety of new refonns. Many faculty and staff are increasingly focused 
on collecting and analyzing data and on the implications for student success. 

The experiences of the Round 1 colleges, described here and in Turning the Tide, repre- 
sent an important piece of the broader story of Achieving the Dream. Yet, the significance of 
the initiative extends beyond the stories of these colleges as recounted in these reports. In gen- 
eral, Achieving the Dream has sought to change a number of spheres, from those concerned 
with public policy to those affecting the general public. The initiative has grown in the decade 
since its founding: Achieving the Dream now stands as an independent organization and has 
expanded to nearly 200 colleges, 15 state policy teams, and more than 100 coaches and advisers 
who participate across 32 states and in Washington, DC. Because the initiative has modified, 
expanded, and codified many of its practices and supports over the years, students in colleges 
that joined the initiative in later rounds of Achieving the Dream may have benefited from the 
lessons learned in the earlier rounds. 

The 26 Round 1 colleges were leaders in this effort, learning alongside the partner or- 
ganizations about institutional change in community colleges. The innovations made by Round 
1 colleges, and their willingness to publicly share their experiences, have pushed the field for- 
ward and helped advance reform efforts in community colleges nationwide. Other system re- 
form initiatives, such as the Developmental Education Initiative and Completion by Design, 15 
have been heavily influenced by lessons learned from the early experiences of the Round 1 col- 
leges. Colleges, policymakers, funders, and researchers across the United States are building on 
these lessons as they think about ways to hasten community college reform. 16 


l5 Janet C. Quint, Shanna S. Jaggars, D. Crystal Byndloss, and Asya Magazinnik, Bringing Developmental 
Education to Scale: Lessons from the Developmental Education Initiative (New York: MDRC, 2013); 
Rutschow et al. (2011). 

16 Quint, Jaggars, Byndloss, and Magazinnik (2013). 
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This report marks the end of a long-term study of the first colleges to join Achieving the 
Dream. The experiences of the Round 1 colleges have demonstrated not only that change can 
occur at the institutional level, but also that substantially improving institution-wide student out- 
comes is much harder than was envisioned. There is still much work to be done to improve the 
prospects of students entering community colleges in the United States, but Achieving the 
Dream has established a solid foundation and a strong network to embark on that work. 
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Chapter 1 

Introduction 


Each year, millions of Americans enroll in community colleges, seeking to develop the skills 
necessary to pursue a career or to transfer to a four-year college. Community colleges serve large 
proportions of nontraditional, low-income, and minority students, 1 yet for most of these students 
academic success remains elusive. Six years after entering community college, almost half of 
first-time students are not enrolled at any institution and have not received a degree or certifi- 
cate. 2 In response to these low rates of student success, shortly after its founding in 2000, Lumina 
Foundation identified community colleges as a “high-need area” that was ripe for systemic re- 
fonn and direct assistance. 3 In 2004, Lumina Foundation and a group of partner organizations 
launched Achieving the Dream, a bold, multiyear national initiative aimed at improving student 
outcomes in community colleges, particularly for low-income students and students of color. 
Seven partner organizations were selected to help Lumina design and operate the initiative. They 
were the American Association of Community Colleges; the Community College Leadership 
Program at the University of Texas; the Community College Research Center (CCRC) at Co- 
lumbia University; Jobs for the Future; MDC, Inc.; MDRC; and Public Agenda. 4 

From the outset, Lumina Foundation and the partners had considerable ambitions for 
Achieving the Dream. The initiative set out to change the culture and practices inside communi- 
ty colleges, as well as external factors that shape institutional behavior, such as public policy, 
research, and public engagement. Colleges that participated in Achieving the Dream received 
multiyear grants and technical assistance to undertake institutional reform directed toward 
building a “culture of evidence” — one in which colleges routinely use evidence to help their 
students succeed academically. This involved leveraging leadership commitment, using data 
and research, and enlisting stakeholders to develop interventions for students to bring effective 
programs to scale and to create an infrastructure for continuous improvement. 

MDRC and CCRC studied the work of the first 26 colleges (called the “Round 1 ” col- 
leges) that participated in Achieving the Dream during the five-year grant period. These colleg- 
es made substantial progress toward realizing this vision. In 201 1, MDRC and CCRC reported 
on the implementation of the initiative and on institution-wide trends in student outcomes at 
these 26 colleges from 2004 through 2009. That report, Turning the Tide: Five Years of Achiev- 
ing the Dream in Community Colleges , 5 found that (1) most colleges made important strides in 


'Provasnik and Planty (2008). 

2 Radford, Berkner, Wheeless, and Shepherd (2010). 

3 Lumina Foundation (2002). 

4 For information on their responsibilities in Achieving the Dream, see Rutschow et al. (2011). 
5 Rutschow et al. (2011). 
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building a stronger culture of evidence, (2) colleges instituted a wide range of strategies to im- 
prove student achievement, though a majority of them remained small in scale, and (3) with a 
few exceptions, institution-wide trends in student outcomes remained relatively unchanged. The 
Round 1 colleges made impressive progress in carrying out the daunting task of transforming 
complex organizations amidst resource and other constraints. The colleges’ experiences high- 
lighted a critical next step in realizing the initiative’s ambition: leveraging the institutional and 
cultural changes facilitated by Achieving the Dream to meaningfully alter the experience of 
large numbers of students. Supporting colleges in their quest to identify, to implement, and to 
bring to scale effective practices remains a key challenge for policymakers, funders, and re- 
searchers committed to improving outcomes for community college students. 

This report, the final publication from MDRC and CCRC on Achieving the Dream, 
builds on Turning the Tide. First, it extends the analyses of institution-wide outcomes to stu- 
dents who were entering Round 1 institutions during the latter period of the five-year implemen- 
tation grants, when colleges were expected to have more fully implemented many of the objec- 
tives of Achieving the Dream. Second, the report explores variation in student outcome trends at 
these colleges and reanalyzes the implementation data in order to identify lessons that could 
inform other institutions that are undertaking reforms. This report is a retrospective study of 
Achieving the Dream as it was implemented between 2004 and 2009 at the first group of col- 
leges to participate, rather than an assessment of the initiative’s current activities and programs. 
The report adds two more years of data to capture outcomes for the cohort that entered college 
in 2009 and four-year outcomes for the cohort that began college in 2007. As with Turning the 
Tide, the student outcome analysis describes college-wide trends. Qualitative implementation 
data that were analyzed in Turning the Tide are reanalyzed in this report; no new implementa- 
tion data were collected. 

A thorough, detailed analysis of the colleges’ implementation of Achieving the Dream 
is provided in the Turning the Tide report. 6 A list of publications that examine other aspects of 
Achieving the Dream — including those that feature case studies of participating colleges and 
results from random assigmnent evaluations of Achieving the Dream strategies — is included at 
the end of this report. 

The remainder of this chapter describes the context in which Achieving the Dream was 
launched, the Achieving the Dream model, the Round 1 colleges, the evaluation charge and 
previous findings, and the contents of this report. 


6 Tuming the Tide is available through MDRC’s Web site, www.mdrc.org. 


2 



Community Colleges and the Achieving the Dream Model 

Traditionally focused on increasing access to postsecondary education, particularly for low- 
income students, community colleges began turning more attention to improving the academic 
success of their students in the early 2000s. 7 Most of these institutions, however, had neither the 
capacity nor the incentives to gather infonnation about the performance of students over time 
and the possible barriers to their success. 

At the time Achieving the Dream was launched, most community colleges across the 
country gathered data on students through enrollment fonns, placement tests, and academic 
transcripts. Because these colleges typically derive their funding based on enrollment numbers, 
they were adept at compiling infonnation on the number and characteristics of students attend- 
ing the college overall and of students in specific academic programs. Community colleges also 
document the number of certificates and degrees conferred each year. 8 It was much less com- 
mon, however, for colleges to track students over time to see whether they were staying en- 
rolled and making steady progress toward attaining a degree. In addition, community colleges 
rarely performed more detailed analyses to determine whether some groups of students experi- 
enced more difficulty than others in completing courses or in attaining degrees. 9 

In this context, the vision of Lumina and of the Achieving the Dream founding partners 
represented a significant, and important, challenge. First, participating colleges were asked to 
use data in a different way — as a tool to determine whether students were staying in school and 
meeting other critical benchmarks. The initiative also asked colleges to break down data in or- 
der to investigate whether there were “achievement gaps” among some segments of the popula- 
tion, such as students of color and low-income students. Beyond this, Achieving the Dream 
aimed to foster fundamental changes in the culture and operations of community colleges, with 
the ultimate aim of increasing the academic success of these population segments. Participating 
colleges were tasked with trying to improve five key outcome measures: 

• Completion of developmental courses and progress to credit-bearing courses 

• Completion of gatekeeper (introductory college) courses in English and math 

• Completion of attempted courses with a grade of “C” or better 

• Persistence from semester to semester and from year to year 

• Attainment of a college credential 


7 Rutschow et al. (2011). 

8 Brock et al. (2007). 

9 Morest and Jenkins (2007). 
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To work toward this goal, Lumina and the founding partners sought to bring about a 
process of institutional change by providing multiyear grants and technical assistance through 
Achieving the Dream, combined with colleges’ own investments. This process centered on 
building a “culture of evidence”: the partners theorized that creating systemic institutional 
change would ultimately lead to institution-wide improvements in the five outcome measures. 
In particular, Achieving the Dream was envisioned as a means to engage colleges in systematic, 
data-driven decision making. This was to be accomplished by enlisting broad groups of faculty, 
staff, and others to identify needs, to implement interventions, and to continuously improve 
practices based on evaluation results. This approach, in turn, would give colleges a better idea of 
the performance levels of students and of how the reforms enacted might be affecting students. 
This information would help provide increasing numbers of students with more effective pro- 
grams and services over time, ultimately leading to improvement in the five key outcomes listed 
above. The initiative outlined five steps to guide participating institutions in this process of 
change, as shown in Figure 1.1: 

• First, leaders commit to making policy changes and to allocating resources 
to support efforts aimed at increasing student success. 

• Second, colleges use data to ascertain how students are perfonning, to iden- 
tify groups that may need extra support, and to prioritize actions. 

• Third, colleges engage faculty, staff, and other stakeholders in the use of 
data to develop intervention tools to address problems that they identify as 
priorities. 

• Fourth, colleges implement and evaluate strategies and use results to make 
decisions about whether to expand or refine the strategies. 

• Finally, colleges build an infrastructure to support continuous, systemic 
improvement by institutionalizing effective policies and practices. This is 
based on the premise that program review, planning, and budgeting are driv- 
en by evidence of what works best for students. 

Thus, the colleges were asked to make significant changes in a relatively short period 
and, considering the overall operating budget that was available, with relatively limited addi- 
tional funds. The five steps outlined above build on and reinforce one another. Theoretically, 
colleges needed to master the earlier steps in order to complete the later steps and to ultimately 
engender large-scale institutional change. Moreover, this process was envisioned as iterative. 
For example, after bringing to full scale an intervention strategy that had been found to be suc- 
cessful, colleges would repeat the initial steps of the process to identify and to address new 
problem areas. Achieving the Dream also emphasized this process as a means to overcome 
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Figure . 

Achieving the Dream’s Five-Step Process for Improving Student Success 
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Implement, 
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Improved student outcomes and reduced 

achievement gaps among subgroups of 

students based on five indicators: 

■ Completion of developmental courses and 
progression to credit-bearing courses 

■ Completion of gatekeeper courses, particularly 
first college-level or degree-credit courses in 
math and English 

■ Completion of attempted courses with 
a C or better 

■ Persistence from term to term and year to year 

■ Attainment of credentials 


SOURCE: Field Guide for Improving Student Success (2009), p. 1 1, from the Achieving the Dream: Community 
Colleges Count initiative. Copyright 2009 by Lumina Foundation Reprinted with permission. Available 
online at www.achievingthedream.org/docs/Field_Guide_for_Improving_Student_Success.pdf. 
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achievement gaps, particularly among students of color and low-income students. The initiative 
encouraged colleges to analyze the perfonnance of these students and to develop interventions 
aimed at increasing their success. 

Lumina Foundation and the partner organizations also developed a plan to coordinate 
efforts to target policymakers, researchers, individuals, and national organizations that repre- 
sent, support, and study community colleges. In the area of public policy, the plan called for 
encouraging state lawmakers and higher education officials to adopt policies that could help 
community colleges to serve their students more effectively, and to build consensus on state 
reporting requirements and measures of student success. In regard to research, the plan called 
for the investigation of factors that contribute to or that impede the success of community col- 
lege students. Also explored were factors that enhance the effectiveness of programs designed 
to boost achievement. On an individual level, the plan aimed to increase awareness of the work 
community colleges do, and to engage people in supporting the mission of Lumina Foundation 
and of the partner organizations. Finally, among national organizations, the goal of the plan was 
to promote regular meetings, infonnation sharing, and collaboration. 


Achieving the Dream Round 1 Colleges 

From the beginning, Lumina Foundation and its partners envisioned Achieving the Dream as a 
national initiative that would involve many states and institutions. Rather than spread resources 
too thinly, however, Lumina and the partners agreed to concentrate on particular regions of the 
country and to expand gradually. They also believed that it would be advantageous to create a 
“critical mass” of institutions within certain states that could work together on state budget is- 
sues and other legislative priorities. Florida, New Mexico, North Carolina, Texas, and Virginia 
were chosen for the first round of Achieving the Dream efforts, largely because they were per- 
ceived as having favorable climates for policy change, including stable funding and high-level 
support for community colleges. Particular emphasis was placed on the promotion of an equity- 
based agenda and the inclusion of institutions with high concentrations of low-income students, 
of students of color, and of nontraditional students. 

Twenty-six colleges in these states — known as the “Round 1” colleges — were select- 
ed to participate in the initiative beginning in 2004. 10 Lumina was willing to invest in a diverse 
group of colleges to leam how enduring reform could be successfully accomplished, and the 
Round 1 colleges were ready, at varying levels, to take on reform work. Turning the Tide re- 


10 Community colleges in these states that served large proportions of students of color and low-income 
students were invited to apply, and 27 colleges were selected based on their proposals. One college withdrew 
after the first year, leaving 26 colleges in Round 1 . Some multicampus districts, such as Alamo Community 
College District, are counted as one college. For more information about the selected colleges, see Brock et al. 
(2007) and Rutschow et al. (2011). 
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ported on the implementation and student outcome trends at the Round 1 colleges and districts 
listed in Table 1.1, and this report also focuses on those colleges. 

The Round 1 colleges are diverse in size, location, and student characteristics. They 
range from Houston Community College, with a full-time equivalent (FTE) enrollment of over 

38.000 students in 2009-2010, to Martin Community College, with an FTE enrollment of under 

1.000 students. The 26 colleges are located in both large and midsized cities, as well as in sub- 
urbs and small towns. White students make up a majority, or plurality, of students at most of the 
institutions, but nearly all of the colleges enroll substantial numbers of African-American, His- 
panic, and Native American students. 1 1 

Achieving the Dream provided financial support and technical assistance to support 
Round 1 colleges. The schools received an initial year-long planning grant of $50,000 each, fol- 
lowed by annual grants of $100,000 for four years. 12 Additionally, they received support from 
two outside consultants: a coach, who helped them to set priorities, to build consensus, and to 
implement strategies for improvement, and a data facilitator, who helped with data collection, 
analysis, and interpretation. During both the planning year and the first year, these consultants 
visited their assigned colleges four times, for a total of about 12 days. During their final year, 
the consultants made one three-day visit. Achieving the Dream also sponsored a kick-off con- 
ference and annual conferences in which teams from participating colleges convened to share 
ideas and lessons. 

In analyzing the work and progress of the Round 1 colleges, it is important to note that 
Round 1 schools joined Achieving the Dream as it was still evolving. Since its inception in 2004, 
the initiative has expanded, modified, and codified many of its practices and supports. Round 1 
colleges received less intensive versions of these conventions during the early phases of the im- 
plementation. As such, their progress should be seen as a glimpse into the initiative’s influence 
during its early development, when its practices and supports were still being solidified. 


Evaluation Charge 

MDRC and CCRC were charged with studying the implementation of Achieving the Dream, as 
well as institution-wide trends in student outcomes at the 26 Round 1 colleges during the five- 
year grant period. 


n For more detail on the characteristics of individual colleges when they entered Achieving the Dream, see 
Brock et al. (2007). 

12 Colleges joining Achieving the Dream from 2005 to 2010 sometimes received smaller amounts of mon- 
ey over a shorter period, depending on the funders that supported them. The majority received the same 
amount as did Round 1 colleges. As of 2010, to support the sustainability of Achieving the Dream as an inde- 
pendent nonprofit entity, colleges were required to pay to participate, though some institutions did find spon- 
sors. For more information, see Rutschow et al. (2011). 
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Achieving the Dream: Community Colleges Count 
Table 1.1 

Round 1 Colleges, Academic Years 2004-2005 and 2009-2010 


State/College 

Location 

2004-2005 

Enrollment" 1 

2009-2010 

Enrollment" 1 

Florida 




Broward College 

Ft. Lauderdale 

22,540 

27,862 

Flillsborough Community College 

Tampa 

16,157 

19,397 

Tallahassee Community College 

Tallahassee 

9,819 

11,550 

Valencia Community College 

Orlando 

20,727 

29,872 

New Mexico 




Central New Mexico Community College 

Albuquerque 

14,955 

19,200 

New Mexico State University-Dona Ana 

Las Cruces 

3,656 

5,412 

Santa Fe Community College 

Santa Fe 

2,307 

1,948 

Southwestern Indian Polytechnic Institute 

Albuquerque 

699 

2,771 

University of New Mexico-Gallup Campus 

Gallup 

1,814 

729 

North Carolina 




Durham Technical Community College 

Durham 

4,028 

4,044 

Guilford Technical Community College 

Jamestown 

7,513 

12,393 

Martin Community College 

Williamston 

717 

597 

Wayne Community College 

Goldsboro 

2,911 

3,315 

Texas 




Alamo Community College District Central Office b 



Northwest Vista College 

San Antonio 

3,448 

15,828 

Palo Alto College 

San Antonio 

3,416 

9,274 

San Antonio College 

San Antonio 

9,027 

27,935 

St. Philips College 

San Antonio 

5,521 

12,776 

Brookhaven College 

Farmers Branch 

6,629 

7,431 

Coastal Bend College 

Beeville 

2,893 

2,820 

El Paso Community College 

El Paso 

17,084 

18,848 

Galveston College 

Galveston 

1,583 

1,440 

Flouston Community College System 

Flouston 

26,341 

38,035 

South Texas College 

McAllen 

11,478 

17,924 

Southwest Texas Junior College 

Uvalde 

3,488 

3,860 

Virginia 




Danville Community College 

Danville 

2,470 

2,854 

Mountain Empire Community College 

Big Stone Gap 

1,846 

2,136 

Patrick Flenry Community College 

Martinsville 

2,242 

2,629 

Paul D. Camp Community College 

Franklin 

821 

982 

Tidewater Community College 

Norfolk 

15,078 

21,507 


SOURCE: U.S. Department of Education, National Center for Education Statistics, Integrated 
Postsecondary Education Data System (IPEDS). 


NOTES: a Full-time equivalent 12-month enrollment numbers are shown. 

b Four colleges in the Alamo Community College District are participating in Achieving the 
Dream. The district is the recipient of the Achieving the Dream grant. 






With respect to the student outcome measures, Achieving the Dream partners had 
high hopes of “moving the needle” at participating colleges. Some of the partners thought the 
colleges’ cultural and institutional changes would affect student outcomes in a relatively short 
time frame; others believed it was a long-term process. As the initiative represented a funda- 
mental shift in practices for many colleges, some of the partners believed that full implementa- 
tion could take longer than the five years of grant funding, carrying implications for the ex- 
pected time frame of improvement in student outcomes. Moreover, even if the Achieving the 
Dream model were effective in improving short-term outcomes, external factors unrelated to the 
initiative might impede measurable improvements. Some partners invoked the metaphor of 
turning a ship to describe the work of the initiative: while they firmly believed it would set 
community colleges on a better course, they recognized that changes would likely occur gradu- 
ally and might not be apparent for several years. 

Overall, Achieving the Dream was envisioned as a large-scale process designed to 
achieve systemic, institution- wide change. In line with this vision, MDRC and CCRC were 
asked to analyze student outcome trends at the institutional level. That is, for each cohort, they 
were to describe college-wide outcomes, as well as developmental course completions for stu- 
dents who placed into developmental education. Student outcome trends in particular sub- 
groups, such as those defined by race, ethnicity, and receipt of a federal Pell grant (a proxy for 
low-income status), were also identified for analysis. This approach reflects the ambition and 
scope of the initiative: the Achieving the Dream outcomes analysis was designed to take a 
broad view of institutional change and to examine the progress of entire cohorts of students 
attending college. 

The Achieving the Dream model involved implementing and ultimately bringing to 
scale interventions that would directly affect students. It also required laying the groundwork for 
a culture of student success and continuous improvement, for example, by involving higher- 
level administrators and spurring reforms in institutional research departments. While the initia- 
tive emphasized the importance of disaggregating data, the partners believed that the process as 
a whole would lead to stronger interventions that together would positively affect college-wide 
student outcomes. They also believed that the cultural and institutional changes themselves 
would engender changes in student outcomes. Consequently, the partners collectively decided 
that broad, institution-wide outcomes would best reflect these outcomes and that these should be 
used for the evaluation. 

Scaling, however, proved particularly difficult within the five-year grant period. As a 
result, many students at Round 1 institutions were unlikely to experience the intervention re- 
forms undertaken during this time frame. Consequently, institution-wide trends may not reflect 
the impact of these refomis on the smaller group of students who did experience the reforms. 
For this report and for Turning the Tide , however, the partners’ early decisions about using in- 
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stitution-wide student outcome measures meant that additional data on individual students were 
not collected, making it impossible to identify and analyze outcomes for subgroups such as stu- 
dents who participated in Achieving the Dream interventions. 


Previous Findings on Achieving the Dream 

Released in early 2011, Turning the Tide examined the efforts of Round 1 colleges to build a 
culture of evidence during the five-year grant period (2004-2009), as well as trends in student 
outcomes during that time across the 26 colleges. Overall, the report found that most Round 1 
colleges made important strides in building a culture of evidence — that is, they had strength- 
ened the components of the initiative’s institutional improvement model. Most colleges attribut- 
ed this progress to Achieving the Dream as well as to other forces, such as accreditation pro- 
cesses and other grants. 13 While colleges varied in their implementation of the model, many 
enhanced their leadership commitment to student success, increased their research capacity, and 
implemented a wide range of strategies aimed at improving student achievement. 

• Altogether, 21 of the 26 Round 1 colleges (81 percent) improved their cul- 
ture of evidence over the course of their five-year participation in Achieving 
the Dream. 

• By the end of their five-year participation, 1 1 Round 1 colleges (42 percent) 
had implemented most of the practices associated with a strong culture of ev- 
idence. These colleges featured broad-based involvement of college adminis- 
trators, of faculty, and of staff; strong institutional research departments that 
produced readily understandable reports on student achievement; regular 
evaluations of interventions to improve student success; and attention to scal- 
ing up program strategies that helped students to become more successful. 

• Ten colleges (38 percent) had instituted many aspects of the suggested im- 
provement process of Achieving the Dream by the end of the five-year peri- 
od, though not to the same degree as those described above. 

• At the completion of the grant period, five colleges (19 percent) were still 
struggling to implement a number of the recommended practices by spring 
2009. Weak institutional research departments hindered the ability of most of 
these schools to institute a broad data-driven culture. 


13 Findings summarized in this paragraph and the next three are from Rutschow et al. (2011). 
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Round 1 colleges also implemented a large number of strategies under the auspices of 
Achieving the Dream. Most of these involved the implementation of programs and services de- 
signed to improve students’ success. Characteristics of the strategies are detailed below. 

• While the strategies ranged widely, from low-intensity orientation activities 
to more intensive curricular refomis, a large majority were designed to in- 
crease academic and social support systems; only about one-fourth changed 
the content and delivery of classroom instruction itself. 

• Nearly half of the strategies targeted developmental education students, and 
one-third focused on students in their first year of college. 

• Nearly all of the colleges succeeded in expanding at least one strategy to 
reach at least 25 percent of its intended target population. However, despite 
the institutions’ notable efforts to scale up their programs and services, the 
majority of strategies reached less than 10 percent of the intended target pop- 
ulations. This was particularly true when the strategies involved the kind of 
intensive contact that might be expected to influence students’ performance 
to a meaningful extent. The benefits of promising interventions, therefore, 
were frequently extended to only a fraction of students. 

In summary, Turning the Tide documented the important institutional and cultural 
changes that took place at Round 1 colleges during the Achieving the Dream grant period. The 
research also revealed a number of areas for improvement, including (1) the need for an in- 
creased focus on faculty and staff engagement, particularly among adjunct or part-time faculty; 
(2) further definition of efforts to reduce achievement gaps by race, by ethnicity, and by income 
level; and (3) a more systematic effort to help colleges expand and institutionalize their efforts 
to improve student success. In particular, a key task remained at the end of the five-year period: 
bringing these system-level changes to the student level in order to change the experience of 
large numbers of students in a significant way. As noted above, relatively small proportions of 
students experienced the strategies that colleges implemented as part of Achieving the Dream. 

Perhaps unsurprisingly, then, average college-wide trends in student outcomes across 
Round 1 colleges remained relatively unchanged from pre-initiative to post-initiative cohorts, 
with a few exceptions. Though there were modest improvements over time in the rates for com- 
pletion of gatekeeper English and in overall course completion, other outcomes remained sub- 
stantially the same in the years immediately before and after Achieving the Dream began. As 
with implementation, there was considerable variation in student outcome trends across Round 
1 colleges; some schools improved on certain outcomes over the five years and some declined. 
As mentioned, since these student outcome analyses relied on institution-wide, descriptive 
fiends over time, they cannot address the impact of Achieving the Dream on participating col- 
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leges, or any of the outcome trends for those students specifically exposed to any interventions 
implemented as part of Achieving the Dream. 

MDRC and CCRC also studied the effects of Achieving the Dream on the six colleges 
in Washington State that joined the initiative in 2006 as part of the third round of colleges. As 
with Round 1 colleges, most of the institutions in Washington laid the groundwork for systemic 
reforms. However, many strategies for improving student success continued to reach relatively 
small proportions of students. Likewise, the analysis of trends in student outcomes did not re- 
veal clear patterns of improvement for the Washington State colleges since they joined Achiev- 
ing the Dream. 14 


Report Organization 

The implementation data in Turning the Tide cover the full five-year grant period (2004-2009). 
However, as a result of the timing of data reporting, processing, and analysis, the student out- 
come data reported therein did not capture the outcomes of students who entered college toward 
the end of the grant period. To include outcomes for these students — and thus align the time 
frame of the outcome data with the implementation data — this report adds two more years of 
student outcomes data to the analysis of Round 1 colleges, examining the period through 2011. 
This more extended duration allows for the analysis of two-year outcomes that include the co- 
hort entering in 2009, when changes in Achieving the Dream were likely to be more established 
and students entering college were thus most likely to experience such changes. 15 However, 
since the analysis reported in Turning the Tide found that, as of 2009, large proportions of stu- 
dents were not yet directly affected by the initiative, notable changes may not be apparent even 
with an additional two years of follow-up data. 

Nevertheless, Achieving the Dream has emphasized the importance of collecting and 
analyzing student outcomes data. This report was conceived to analyze these outcomes using 
follow-up data beyond the data analyzed in Turning the Tide. Doing so provides an important 
perspective on the success rates for community college students across diverse institutions. The 
additional data permit the tracking of two-year outcomes for five successive cohorts of students, 
and four-year outcomes for three successive cohorts, all of whom entered college after the 
Round 1 institutions began implementing Achieving the Dream. 

Chapter 2 discusses the analyses of these data, in addition to student outcome data pre- 
viously collected for the three cohorts of students who entered college before the initiative. It 


14 Jenkins, Wachen, Kerrigan, and Mayer (2012). 

l5 The additional data also enable the analysis of four-year outcomes through the cohort beginning college 
in 2007. Turning the Tide analyzed two-year outcomes, through the cohort that started in 2007, and did not 
examine four-year outcomes. 
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draws on student records data that the colleges submitted to the Achieving the Dream data- 
base, 16 in order to track student perfonnance over time on the five key outcome measures. The 
analyses in this chapter and throughout the report are not causal — they do not evaluate the im- 
pact of Achieving the Dream — but rather describe the colleges’ perfomiance over time with 
respect to these key outcome measures. 

Chapter 2 also explores the variation in student outcome trends first documented in 
Turning the Tide. It leverages the additional two years of student outcome data to identify col- 
leges that consistently improved more than others in order to examine their experiences more 
fully. As with Turning the Tide, the descriptive research design cannot address the effects of 
Achieving the Dream, nor can the institution-wide trends provide insight into outcome trends 
for students who received Achieving the Dream intervention services. 

The descriptive analysis in Chapter 3 is based on implementation research conducted on 
Round 1 colleges during the five-year grant period. This chapter examines those implementa- 
tion themes and patterns at the colleges identified in Chapter 2 that showed the greatest con- 
sistent improvements in student outcomes across multiple indicators. While this evaluation 
cannot explain why some colleges saw greater improvements than others, or whether the im- 
provements were attributable to the institutions’ involvement in Achieving the Dream, Chapter 
3 recounts some interesting stories of implementation successes, challenges, and lessons 
learned from these colleges. (More detail on the implementation of the initiative at Round 1 
colleges can be found in Turning the Tide.)' 1 

Chapter 4 summarizes and synthesizes the findings and offers recommendations and 
considerations regarding more systematic integration of program development and research in 
community colleges. 


This Report in the Context of Achieving the Dream 

The experiences of the Round 1 colleges described in the following chapters, and in Turning the 
Tide, represent an important piece of a broader story. The meaning and import of Achieving the 
Dream extend beyond these experiences, making it useful to consider this report in the context 
of the full initiative. 

While this evaluation is limited to efforts undertaken inside community colleges, 
Achieving the Dream sought to influence a number of spheres, with implications for a range of 
areas, from those concerned with public policy to those affecting the general public. Achieving 
the Dream has grown in the decade since its founding. In 2010, it made the transition from be- 


16 JBL Associates, a higher-education consulting firm under contract with CCRC, managed the database. 
17 Rutschow et al. (2011). 
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ing managed by a group of partner organizations to achieving status as an independent nonprofit 
organization: Achieving the Dream, Inc. The initiative has expanded to nearly 200 other schools, 
though this report cannot address the experiences of these other colleges. Achieving the Dream 
has expanded, modified, and codified many of its practices and supports over the years. Conse- 
quently, students in colleges that joined in later rounds may have had different experiences and 
may have benefited from lessons learned from the involvement of those who took part in the 
early rounds of the initiative. 

Although the implementation data for this report cover only the five-year grant period 
of Achieving the Dream, for the most part the process set in motion by the initiative did not end 
with the grant. In 2009, most Round 1 colleges reported that the progress they made in building 
a culture of evidence and in developing intervention strategies would continue. Together, the 
capacity they built, the culture they fostered, and the strategies they launched as part of Achiev- 
ing the Dream may serve as an important foundation for efforts to improve student outcomes in 
the years to come. Some colleges are building on the work they accomplished in the Achieving 
the Dream project by participating in other large-scale initiatives. Fifteen colleges that took part 
in the early stages of Achieving the Dream received funding from one such program, the De- 
velopmental Education Initiative, to develop and to scale up developmental education refomis. 
Another effort, Completion by Design, allows participating colleges to design and to implement 
comprehensive reforms to support students through completion of community college. 18 

The experiences of the Round 1 colleges during the five-year grant period of Achieving 
the Dream provide valuable lessons for the field. As with the Achieving the Dream model itself, 
developing and supporting large-scale institutional refomis will likely prove to be an iterative 
process that relies on a continuous and honest assessment of progress and challenges. This re- 
port is a contribution to that effort. It reflects on lessons learned from the first colleges to partic- 
ipate in the initiative and offers recommendations to funders, practitioners, researchers, policy- 
makers, and other stakeholders in an effort to continue working toward improvement. 


18 See Quint, Jaggars, Byndloss, and Magazinnik (2013); www.completionbydesign.org. 
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Chapter 2 

Student Outcomes 


This chapter analyzes trends in student outcome measures. It compares the outcomes for three 
cohorts of students who entered college before the implementation of Achieving the Dream, 
from fall 2002 through fall 2004, to the outcomes for five cohorts who began college after im- 
plementation began, from fall 2005 through fall 2009. The analyses examine six outcomes iden- 
tified by Achieving the Dream: (1) completion of developmental English courses within two 
years, (2) completion of gatekeeper (introductory college) English courses within two years, (3) 
completion of developmental math courses within two years, (4) completion of gatekeeper (in- 
troductory college) math courses within two years, (5) persistence from semester to semester, 
and (6) attainment of college credentials within four years. 

The analyses in this chapter are not causal: they do not evaluate the impact of Achiev- 
ing the Dream. However, the initiative has emphasized the importance of collecting and analyz- 
ing data on student outcomes. This report was conceived to analyze student outcomes using two 
years of additional follow-up data subsequent to the publication of Turning the Tide, 1 an earlier 
report on Achieving the Dream, and to use additional data to examine trends in four-year gradu- 
ation rates. Doing so provides an important perspective on success rates for community college 
students across diverse institutions. The analyses in this chapter describe the colleges’ perfor- 
mance with respect to the six key student outcome measures identified above. They illustrate 
where student outcome measures have improved and where they have not. 

The key findings are: 

• The average student outcome trends for Round 1 colleges remained 
largely unchanged over time, with one exception. As in Tinning the Tide, 
the average student outcome fiends for gatekeeper English showed continued 
gains. Outcome fiends in developmental English and math, in gatekeeper 
math, in persistence, and in four-year completion rates showed little or no 
improvement. Because these descriptive analyses cannot account for the 
many changes in the external environment that occurred during this period, 
these findings cannot be interpreted as being caused by Achieving the 
Dream. 

• The colleges exhibited a wide range of gains and declines for each stu- 
dent outcome measure. However, changes in student outcomes were not 


Rutschow et al. (2011). 
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found to be systematically related to practices at the colleges or to other insti- 
tutional factors. Over the evaluation period, some colleges made gains on in- 
dividual indicators and others declined. Generally, though, colleges that 
made the greatest gains on one indicator did not do so on other indicators. 

• Three colleges stood out for their gains on multiple student outcomes. 

These institutions consistently ranked among the top five. As measured by 
subtracting the average for the outcome for cohorts who entered college in 
the pre-initiative period (2002-2004) from the average for cohorts who began 
college after the start of implementation (2005-2009), these schools showed 
the greatest gains on at least three student outcome measures. 


Methodology 

This chapter uses student data from the Achieving the Dream database, 2 which includes out- 
comes for all degree- or certificate-seeking students who entered the institutions for the first time 
each fall. The data include student-level characteristics but do not identify students’ participation 
in different strategies implemented as part of Achieving the Dream. Consequently, this report 
examines college-wide outcomes and developmental course completions for students who placed 
into developmental education, as well as outcomes for particular subgroups, such as those de- 
fined by receipt of a federal Pell grant (a proxy for low-income status), by race, and by ethnicity. 

The analyses in this chapter are descriptive. This chapter does not seek to determine 
whether the initiative as a whole had an effect on student perfomiance, nor whether specific 
Achieving the Dream strategies led to changes in student outcomes. Such conclusions require 
rigorous causal research designs, such as randomized controlled trials or designs that use a valid 
comparison group, neither of which are well-suited to the evaluation of Achieving the Dream. 3 
Moreover, as documented in Turning the Tide, most Achieving the Dream strategies did not 
reach large proportions of students by the time the last cohort in this analysis entered college. 
Consequently, even if the strategies that colleges implemented did affect smaller proportions of 
students, the potential effects on those students’ outcomes would probably not be apparent in 
the college-wide trends described in this report. Since the student-level data in the Achieving 
the Dream database do not identify which students were exposed to Achieving the Dream strat- 
egies, the analyses here cannot address trends for these students only. Reform efforts, moreover, 
were typically directed toward smaller, targeted populations, whereas the measures reported 


2 JBL Associates, a higher-education consulting firm under contract with the Community College Research 
Center, managed the database. 

3 For findings from randomized controlled trials of strategies fostered through Achieving the Dream, see 
Visher, Butcher, and Cema (2010); Weiss, Visher, and Wathington (2010); Weissman et al. (2011); Rutschow, 
Cullinan, and Welbeck (2012). 
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here are restricted to institution- and initiative-wide trends, in addition to some subgroup trends. 
Any effects that may have been experienced by the targeted populations could be difficult to 
detect in institutional averages. 

Several other important changes also occurred during this period that might have affect- 
ed students and exerted pressure on community colleges, also contributing to difficulties in iso- 
lating the effects of Achieving the Dream. Notably, the Great Recession started toward the end 
of 2007, and during the course of the study the colleges witnessed large population shifts. For 
example, the average full-time enrollment at Round 1 colleges increased from just over 6,200 
students in fall 2002 to over 8,500 students in fall 2009 — a gain of over 37 percent — with the 
biggest increases occurring in 2008 and in 2009. The rising enrollments were accompanied by 
shifting demographics, with the proportion of minority students gradually increasing over time. 
In addition, state funding for higher education was also cut during this period, driving lower 
per-student revenue. 4 These and other changes, such as financial pressures, may have influenced 
the colleges in a way that affected student outcomes. 

Finally, nine of the 26 Round 1 colleges did not provide data for the full study. There- 
fore, the data provided in this report do not fully reflect student outcomes for all Round 1 col- 
leges for all cohorts. As noted below, the colleges that did not submit all of the data tended to 
have lower student outcome levels in the earlier cohorts for which they did submit data. 


Trends in Student Outcomes 

The analyses in this section of the report show average institutional outcome trends for six ma- 
jor indicators of student success. 

Developmental and Gatekeeper Math Completion 

Figure 2.1 shows the trends in student completion of key math courses. The top panel 
shows the average two-year completion rate for the developmental sequence in math for stu- 
dents referred to developmental math. The vertical lines in Figure 2. 1 and in the two subsequent 
figures represent the 2004 cohort — the last cohort to enroll before the colleges began imple- 
mentation of Achieving the Dream. For the three cohorts that entered college before implemen- 
tation of Achieving the Dream — the 2002 through 2004 cohorts — the developmental math 
completion rate stays very close to 23 percent. For the cohorts that entered college after im- 
plementation of Achieving the Dream had begun — the 2005 through 2009 cohorts — the 


4 Kirshstein and Hurlburt (2012). 
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Achieving the Dream: Community Colleges Count 
Figure 2.1 

Trends in Developmental Math Among Students Referred to Any Developmental 
Math Course and Gatekeeper Math Completions Among All Students, 
2002-2009: Two-Year Outcomes for Achieving the Dream Round 1 Colleges 
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Figure 2.1 (continued) 


SOURCE: MDRC calculations using the Achieving the Dream database maintained by JBL Associates. 

NOTES: Calculations for these figures used institutional means for all available data for sample members in fall 
2002 through fall 2009 cohorts at Achieving the Dream Round 1 colleges. The vertical line shows the last cohort to 
enroll before the implementation of Achieving the Dream. For the developmental math and gatekeeper math 
completion measures, 1 site was excluded from cohort 2002, 2 sites from cohorts 2007 and later, 4 sites from 
cohorts 2008 and later, and 12 sites from cohort 2009 because of insufficient data for 2-year outcomes. 


completion rate also averages around 23 percent. It gradually rises for three cohorts immediate- 
ly following the start of implementation before falling back to pre-initiative levels in the 2009 
cohort, the last cohort for which two-year outcomes are available in these data. 

The bottom panel of Figure 2.1 shows the completion rates for gatekeeper math for all 
students; this trend is also relatively stable. For the 2002 through 2004 cohorts that entered col- 
lege before implementation, the completion rate for gatekeeper math stays close to 14 percent. 
Immediately following implementation, it remains relatively steady around 14 percent, up 
through the 2008 cohort. Although a slight increase in completions is shown for the 2009 co- 
hort, several colleges did not report sufficient data to be included in the 2009 averages. The up- 
ward trend appears to be due primarily to the fact that colleges that did submit sufficient data 
tended to have higher completion rates in gatekeeper math in prior years. 

Developmental and Gatekeeper English Completion 

Figure 2.2 shows the trends in student completion of key English courses. The top panel 
shows the average two-year completion rate for the developmental education requirements in 
English for students who placed into developmental English. The completion rate here is less 
stable than that in developmental math. For cohorts that entered college before implementation 
of Achieving the Dream, the average is close to 36 percent. After implementation, the develop- 
mental English completion rate moves up and down more frequently, but also averages about 
36 percent. Improvements are apparent for some cohorts, but they are offset by declines in other 
cohorts. 


The bottom panel of Figure 2.2 shows the completion rates for gatekeeper English for 
all students. In the pre-initiative period, the completion rate for gatekeeper English shows a 
slight increase, rising to nearly 25 percent. Contrary to other outcomes that remained relatively 
flat, the trend in gatekeeper English shows steady increases after implementation began, as doc- 
umented in Turning the Tide. By the end of the follow-up period, the average completion rate is 
over 30 percent. As with gatekeeper math, a slight increase in gatekeeper English completions is 
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Achieving the Dream: Community Colleges Count 
Figure 2.2 

Trends in Developmental English Among Students 
Referred to Any Developmental English Course 
and Gatekeeper English Completions Among All Students, 
2002-2009: Two-Year Outcomes for Achieving the Dream Round 1 Colleges 
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Figure 2.2 (continued) 


SOURCE: MDRC calculations using the Achieving the Dream database maintained by JBL Associates. 

NOTES: Calculations for these figures used institutional means for all available data for sample members in fall 
2002 through fall 2009 cohorts at Achieving the Dream Round 1 colleges. The vertical line shows the last cohort 
before the implementation of Achieving the Dream. For the developmental English and gatekeeper English 
completion measures, 1 site was excluded from cohort 2002, 2 sites from cohorts 2007 and later, 4 sites from 
cohorts 2008 and later, and 12 sites from cohort 2009 because of insufficient data for 2-year outcomes. 


shown for the 2009 cohort. However, several colleges did not report sufficient data to be in- 
cluded in the 2009 averages. Those colleges had lower outcomes in previous years, which ap- 
pears to skew the average upward. 

Fall-to-Fall Persistence and Four-Year Completion 

Figure 2.3 shows the trends in fall-to-fall persistence and in four-year college comple- 
tion rates for any credential. The top panel shows the average fall-to-fall persistence rate for 
students’ first full year in college. This rate is also relatively stable, hovering around 47 percent 
for the cohorts that entered college before implementation. After implementation, the persis- 
tence rate shows more fluctuations, with slight improvements again offset by declines. 

The bottom panel of Figure 2.3 shows the average rate of college completion in four 
years. The four-year completion rate averages around 14 percent for the pre- implementation 
cohorts, but rises slightly for the post-implementation cohorts. As with other outcomes, the 
slight gain for the 2009 cohort appears to arise largely because several colleges with lower 
completion rates in earlier years did not submit sufficient data for inclusion, thereby skewing 
the average upward. 

Subgroup Analyses 

One of the goals of the Achieving the Dream initiative was to reduce achievement gaps 
by race, by ethnicity, and by income level. As in Turning the Tide, trends for several subgroups 
of students were analyzed to look for differences in improvement patterns over time. The sub- 
groups were defined by receipt of a federal Pell grant (used as a proxy for low-income status), 
as well as race/ethnicity (black, Hispanic, and white). No notable differences in trends on the six 
main outcomes were found after incorporating the additional two cohorts of data collected since 
the release of Turning the Tide. 
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Achieving the Dream: Community Colleges Count 
Figure 2.3 

Trend in Fall-to-Fall Persistence Among All Students, 2002-2009, 
and Trend in Completion of Any Credential Among All Students, 2002-2007: 
Four-Year Outcomes for Achieving the Dream Round 1 Colleges 



Figure 2.3 (continued) 


SOURCE: MDRC calculations using the Achieving the Dream database maintained by JBL Associates. 

NOTES: Calculations for these figures used institutional means for all available data for sample members in the 
fall 2002 through fall 2009 cohorts at Achieving the Dream Round 1 colleges. The vertical line shows the last 
cohort before the implementation of Achieving the Dream. For the fall-to-fall persistence measure, 1 site was 
excluded from cohort 2002, 2 sites from cohorts 2007 and later, 4 sites from cohorts 2008 and later, and 12 sites 
from cohort 2009 because of insufficient data for 2-year outcomes. For the completion of any credential measure, 1 
site was excluded from cohort 2002, 2 sites from cohorts 2005 and later, 5 sites from cohorts 2006 and later, and 12 
sites from cohort 2007 because of insufficient data for the 4-year outcome. 


Exploring Variation in the Outcome Trends 

Although the trends analysis suggests that, on average, initiative-wide student outcomes re- 
mained largely unchanged over time, the analyses in Turning the Tide suggested variation 
across the colleges with respect to gains and declines on many indicators. This variation was 
still evident after adding the two additional years of student outcome data collected for this re- 
port. The variation was analyzed to examine whether theoretically important factors were asso- 
ciated with gains in student outcomes. Data from several sources — including notes from site 
visits to each of the colleges and a survey of faculty and administrators — were coded. The fac- 
tors analyzed included colleges’ leadership strength, alignment between institutional goals and 
reforms, institutional data and research capacity, and professional development training. Chang- 
es in student outcomes were not found to be systematically related to such factors. 5 

Cohort data for 2002 through 2009 were used to determine whether the trends at some 
colleges stood out for gains on multiple outcomes. For each of the six indicators analyzed 
above, colleges were ranked by comparing their average outcomes after the beginning of 
Achieving the Dream implementation with their averages in the pre-initiative period. Colleges 
with the greatest average percentage point gains after implementation were ranked highest, and 
colleges with the smallest gains (or greatest declines) were ranked lowest. 

Figure 2.4 shows how often colleges ranked among the top five institutions with the 
greatest gains on each of the six outcomes. Three colleges figured prominently. One college 
(College A) ranked among the top five on four out of six outcomes. Two other colleges (Col- 
lege B and College C) ranked in the top five on three of six measures three times each. The ma- 
jority of colleges ranked in the top five less frequently, either two times (six colleges), one time 
(eight colleges), or never (twelve colleges). 


5 More detail about these analyses is available in the Appendix. 
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Achieving the Dream: Community Colleges Count 
Figure 2.4 

Frequencies of Colleges Ranking in Top Five 
Across Six Pre-Post Percentage Point Change Outcomes 



Frequency of ranking in the top five across six outcomes 


SOURCE: MDRC calculations using the Achieving the Dream database maintained by JBL Associates. 

NOTES: Calculations for these figures used institutional means for all available data for sample members in fall 
2002 through fall 2009 cohorts at Achieving the Dream Round 1 colleges. Percentage point change is calculated by 
subtracting the preintervention (2002 to 2004) average from the postintervention (2005 to 2009) average. 


Figure 2.5 shows the distribution of all the colleges’ gains or declines on each of the six 
outcomes and identifies where these three colleges (A, B, and C) ranked for each outcome. 
Each point in Figure 2.5 represents the average change for an individual college for each out- 
come. The figure shows that the gains of these three colleges are concentrated in the gatekeeper 
courses, in fall-to-fall persistence, and in four-year completion. 

The first column in Figure 2.5 illustrates the distribution of average changes in develop- 
mental English completion across the colleges. The college with the greatest gains improved an 
average of about 12 percentage points after implementation of Achieving the Dream began com- 
pared with its average in the pre-initiative period. Conversely, the college with the largest de- 
crease fell an average of almost 15 percentage points. Only one of the consistently improving 
colleges — College C — ranked among the top five. College A showed a small average im- 
provement in percentage points in developmental English completion, while College B saw a 
general decrease on average on this outcome measure. 
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Achieving the Dream: Community Colleges Count 
Figure 2.5 

Pre-Post Percentage Point Change for Round 1 Colleges, by Outcome, 
2002-2004 Versus 2005-2009 



—Average □ Overall high-performing college #Other college 


(continued) 



Figure 2.5 (continued) 


SOURCE: MDRC calculations using the Achieving the Dream database maintained by JBL Associates. 

NOTES: Calculations for these figures used institutional means for all available data for sample members in fall 2002 through fall 2009 
cohorts at Achieving the Dream Round 1 colleges. Percentage point change is calculated by subtracting the preintervention (2002 to 
2004) average from the postintervention (2005 to 2009) average. For the developmental English, developmental math, gatekeeper 
English, gatekeeper math, and fall-to-fall persistence measures, 1 site was excluded from cohort 2002, 2 sites from cohorts 2007 and 
later, 4 sites from cohorts 2008 and later, and 12 sites from cohort 2009 because of insufficient data for 2-year outcomes. For the 4-year 
completion measure, 1 site was excluded from 2002, 2 sites from cohorts 2005 and later, 5 sites from cohorts 2006 and later, and 12 sites 
from cohort 2007 because of insufficient data for the 4-year outcome. 
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The experiences of the colleges are similar for the distribution of changes in develop- 
mental math completion, as shown in the second column in Figure 2.5. Again, there is a wide 
range of colleges that improved and those that did not. Of the three colleges, only College C 
ranked among the top five, improving by about 5 percentage points. On average, both College 
A and College B declined slightly, by around 2 percentage points. For the other outcomes — 
gatekeeper English completion, gatekeeper math completion, fall-to-fall persistence, and four- 
year completion — the three colleges appear among the top colleges more frequently; only Col- 
lege C shows a decline on any of the remaining outcomes (for fall-to-fall persistence). 

In gatekeeper English, all three colleges improved by over 10 percentage points, and 
two (A and B) ranked among the top colleges. Likewise, in gatekeeper math, all three colleges 
improved by between 3 and 7 percentage points, and two (A and C) ranked among the top five. 
With respect to fall-to-fall persistence, two of the colleges (A and B) improved by about 5 per- 
centage points and ranked among the top five, while one decreased (C). On the outcome meas- 
ure of four-year completion, all three of the colleges improved, while two (A and B) ranked 
among the top five. 

Figure 2.5 illustrates the wide range of increases and decreases in student outcome 
measures and also shows how these three colleges stand out on multiple outcomes. While these 
colleges demonstrate that consistent improvement is possible, the overall trends in the previous 
section show that, on average, changes in student outcomes were more modest. In addition, as 
mentioned earlier, the improvements at these three colleges were not necessarily caused by — 
or related to — the Achieving the Dream initiative. The next chapter examines the stories at 
these three colleges in greater detail. 


Summary 

It is important to interpret the analyses in this chapter in the context of the colleges’ implemen- 
tation of Achieving the Dream. As noted earlier, few strategies were reaching large proportions 
of students by the time the last cohort entered college, making the absence of notable changes 
unsurprising. Although these analyses incorporate an additional two years of data on student 
outcomes, institutional changes at community colleges may have either been insufficient to 
change institutional trends or may take longer to ultimately reach a large proportion of students. 
Factors external to Achieving the Dream may also have played an important role in student out- 
comes. In any case, the trends analyzed here do show some areas of improvement but they also 
indicate that more work is needed: except for gatekeeper English, the trends in average student 
outcomes remained largely unchanged over time. Analysis of the perfonnance of individual 
schools across multiple outcomes, however, shows that some colleges consistently improved 
more than others. The following chapter describes some of the implementation experiences of 
these colleges. 
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Chapter 3 

Implementation Stories at Selected Colleges 


One of the key goals of Achieving the Dream is to understand the process by which colleges 
implemented new programs and strategies. The initiative has dedicated significant resources to 
provide a better understanding of how colleges have developed and maintained efforts to im- 
prove their data and institutional research functions, to bring pilot programs to scale, and to fa- 
cilitate deeper faculty and staff involvement in the efforts of community colleges to realize insti- 
tutional goals. 

While Achieving the Dream has made important strides over the past 1 0 years, the pre- 
vious chapter shows that improving institution-wide indicators of student success remains a 
challenge. Most of the initiative’s Round 1 colleges attempted to implement various new pro- 
grams designed to address student needs. Yet they also faced obstacles in their attempts to gain 
wide-scale support and buy-in from faculty and staff to develop and sustain these programs 
amid limited resources and personnel. In addition, colleges provided newly implemented pro- 
grams to smaller groups of students than they originally intended. Developing a coherent set of 
strategies for a targeted group of students also proved difficult when institutions had multiple 
priorities and sought to carry out numerous initiatives, many of which competed for limited re- 
sources. 1 

This chapter considers how the three colleges identified at the end of Chapter 2 ap- 
proached such challenges, and it also describes some of their implementation successes with 
specific reform efforts. Implementation data for the three colleges were first analyzed in an ef- 
fort to discern common patterns that might be associated with their success. A clear set of pat- 
terns did not emerge. In fact, the three colleges look fairly distinct from one another in size, in 
execution of strategies, and in implementation of Achieving the Dream, as might be expected 
from the analyses described in Chapter 2 and in the Appendix. Those analyses codified the im- 
plementation data and looked for associations between a wide array of institution-level factors 
and student outcomes, but they did not find any. For these and other reasons that will be de- 
scribed, the improvements in the student outcomes in these institutions cannot be explained by 
the specific practices at the colleges analyzed in this report. Despite these challenges, the diver- 
sity of the colleges’ experiences may present lessons for other community colleges undertaking 
refonns. The main findings from this investigation are: 


Rutschow et al. (2011). 
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• The colleges profiled represent a diverse set of schools with differing 
student contexts, implementation strategies, and experiences in Achiev- 
ing the Dream. While the colleges shared some similar implementation suc- 
cesses and challenges, their experiences during the grant period were more 
different than similar. No one pattern of program reform and implementation 
emerged from the analysis. 

• The colleges made a concerted effort to use multiple strategies with spe- 
cific subgroups of students. After gaining experience with these initial sub- 
groups, the colleges began to expand the use of these practices to reach larger 
groups of students and faculty. 

• The colleges described in this chapter supported multiple individuals in 
leadership roles, regardless of their official positions. In addition, the col- 
leges adopted the refonn process of Achieving the Dream during periods of 
high-level turnover in leadership. At the same time, new leaders emerged at 
multiple levels within the institutions. 

• In one case, the college leveraged its accreditation process to focus on 
and to coordinate its reform efforts. This college coordinated the accredita- 
tion process with Achieving the Dream and used the process to foster early 
buy-in, to motivate its faculty to focus on the success of math students, and to 
become a more data-driven institution. 


Data Sources 

The analyses in this chapter are based on implementation research conducted on the Round 1 
colleges during the five-year Achieving the Dream grant period. Research team members of 
MDRC and of the Community College Research Center visited each college in spring 2006, 
during the first year of implementation, and again during the fall 2008 and spring 2009 tenns — 
the last year of the Achieving the Dream grant. The visits lasted two to three days and included 
interviews with senior administrators, institutional research staff, the Achieving the Dream co- 
ordinator, and faculty members and student services staff involved in their institution’s efforts to 
implement the initiative. Researchers followed a standard protocol across the 26 colleges to en- 
sure that people in comparable positions were asked similar questions, and that measures of the 
culture of evidence and of adherence to the five-step process of institutional improvement were 
being documented at each college. 

In addition to the field visits, data on the colleges’ progress in implementing the initia- 
tive’s principles and strategies were obtained from internal documents related to Achieving the 


30 



Dream, including annual reports submitted by the colleges and reports submitted by Achieving 
the Dream coaches and data facilitators. No new implementation data were collected since the 
2009 field visits to the colleges, as the central analysis for this report focused on student out- 
comes for cohorts who entered college during the original Achieving the Dream grant period. 
The existing data, however, were re-coded and reanalyzed along with the additional data on 
student outcomes, in an effort to provide new insights into practices at the identified colleges. 


The Colleges 

The three colleges identified in Chapter 2 and analyzed in this chapter are more diverse than 
they are similar, both in their institutional contexts and in the approach they adopted to imple- 
ment strategies and principles related to Achieving the Dream. These institutions range in size 
from small rural colleges to a large multicampus institution located in a metropolitan area. The 
colleges are located in three different states and have distinctive student populations, ranging 
from primarily Hispanic to racially and ethnically diverse. Similarly, while these schools were 
instructed to focus on the initiative’s principles — such as increasing data use and engaging 
more faculty and staff — each executed implementation approaches with their unique institu- 
tional contexts in mind. The following sections describe the three institutional contexts in which 
these Achieving the Dream efforts were carried out. 

College A 

College A is a large multicampus institution located in an urban region in the Southeast. 
It serves a diverse population of over 10,000 students who are primarily of white, African- 
American, and Hispanic ethnicities. Before implementation of Achieving the Dream, faculty and 
staff were concerned about the increasing lack of preparedness among incoming students in addi- 
tion to low retention of rates in developmental courses: nearly 8 in 10 students who enrolled in 
developmental English or math courses did not successfully complete the developmental English 
or math sequence. Consequently, College A’s Achieving the Dream efforts focused on providing 
extra support to students in need of remediation, particularly those required to take college pre- 
paratory courses in all three developmental education areas (reading, English, and math). 

When the initiative began, the college was going through many changes, including the 
development of a new institution-wide education plan and of strategies to target underserved 
students of color. In addition, there was turnover in high-level leadership positions. At the same 
time, College A sought to cultivate greater collaboration across different departments, and it 
took steps to increase the use of data and institutional research functions to support institutional 
goals and strategies. 
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College B 

College B is a midsize institution located in a sprawling suburban region in the South- 
west that serves between 2,000 and 10,000 students. It joined Achieving the Dream with a goal 
to improve student success in developmental mathematics. Another objective was to improve 
success rates for Hispanic students, a population that makes up a substantial proportion of all 
students enrolled. 

Like most community colleges, the majority of students who enroll at College B are not 
prepared for college-level work and need remediation. However, in interviews, administrators 
noted that many such students did not typically seek out additional academic assistance: few 
students utilized the college’s computer lab or tutoring center. As part of its Achieving the 
Dream efforts, the college attempted several pilot strategies that were created to make academic 
support more accessible both inside and outside of class, especially for students struggling 
through developmental math courses. These interventions provided in-class tutoring, software 
to explain and to reinforce content, and refresher workshops. 

The college also wanted to increase its data collection and communication capabilities, 
particularly among faculty. This was partly motivated by a desire to use data to inform devel- 
opmental education practices: because the college already had a well-resourced institutional 
research department that conducted large volumes of data processing, these goals were directed 
mainly at expanding and enhancing these capabilities among faculty and staff who worked di- 
rectly with students. 

College C 

College C is also a midsize college with multiple campuses that is centered in a rural 
region of the South. A majority of students at College C are Hispanic and attend school part 
time. Similar to those in Colleges A and B, students who enroll in College C are often under- 
prepared academically and require developmental education courses. When the college joined 
Achieving the Dream, administrators reported that students in developmental math courses had 
the lowest performance rates across the student population. 

College C entered Achieving the Dream after developing goals that were focused on 
improving student learning outcomes and on strengthening the college’s institutional effective- 
ness. The institutional research department was thinly staffed and lacked the capacity for robust 
gathering and analysis of qualitative data. This college coordinated its Achieving the Dream 
efforts with its accreditation goals. For example, as part of the accreditation process, the college 
identified two priorities — improving institutional data capacity and student performance in 
gatekeeper (introductory college) math courses — that aligned well with its Achieving the 
Dream goal of improving students’ math performance. In order to carry out this objective, the 
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college developed a supplemental instruction program to give students additional support and 
extra time with math content. 


Differences in the Profiled Colleges 

As noted, the three colleges profiled in this chapter had a number of diverse attributes that make 
it difficult to discern larger patterns based on their experiences. These differences may be at- 
tributed to the distinct approaches that each college took to implement Achieving the Dream 
policies and practices, or to differences of procedures that each institution typically endorsed 
based on their overall structures and operational systems. Nonetheless, these differences provide 
further insight into the varied institutional experiences that community colleges may have, even 
when they attempt to implement a similar vision oriented toward improving student success. 

First, Colleges A, B, and C varied in the strength of their implementation of Achieving 
the Dream. College A had a relatively strong culture of evidence, a heavily engaged faculty and 
service staff, and a very strong institutional research staff that grew even stronger during the 
course of the initiative. Conversely, at the beginning of Achieving the Dream, both College B 
and College C had difficulty implementing the initiative. They encountered a number of barri- 
ers, such as leadership turnover and challenges clarifying the goals, vision, and focus of the ini- 
tiative. ft was only in the final years that these schools were able to make more substantial pro- 
gress. However, while each of these colleges made some strides in affecting program-level 
change, all three were still facing challenges in relation to their strategic management and vision 
at the end of their first five years in Achieving the Dream. For the most part, the development of 
a larger, institution- wide strategic plan had been hampered by the colleges’ continuous leader- 
ship turnover, as the presidents and higher-level administrators were often responsible for de- 
veloping and enacting this vision. 

Second, as described in Chapter 2, the colleges also tended to differ on the outcome 
measures on which they saw improvements. There was no indicator on which all three schools 
ranked among the top colleges; in other words, in an area where one or two colleges ranked 
highly, the third did not, and in three out of six of the indicators, at least one college actually 
decreased. In addition, despite the focus of all three colleges on developmental education, only 
College C ranked among the top in improvement of developmental education outcomes. The 
improvements at Colleges A and B were centered on gatekeeper courses, persistence, and grad- 
uation rates. At these colleges, increases in student outcomes sometimes began earlier than the 
institution’s efforts to implement policies and practices for improving success. For instance, 
gains in student outcome were seen early on in Colleges B and C, where implementation pro- 
gress was slower, while College A saw gains later. These disconnects between outcome gains 
and implementation — and the inconsistent pattern of improvements for the targeted subgroups 
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— further underscore the challenge of connecting the increased achievements of students with 
the reform efforts of colleges. 

Third, though all of the colleges used similar, multi-intervention approaches to target 
developmental education students, the specific interventions that were implemented tended to 
vary. For example, Colleges B and C were heavily focused on supplemental instruction, while 
College A concentrated more on intensive case management. In addition, other strategies that 
were implemented at these schools differed, with College A focused on learning communities 
and early registration; College B on preparing students for placement assessment tests and tech- 
nology-aided instruction; and College C on a revised developmental education curriculum, a 
summer bridge program, and technology-aided instruction. Finally, the timing of implementa- 
tion differed across the schools: while one school implemented a number of different interven- 
tions at the beginning of Achieving the Dream, the other two schools implemented most inter- 
ventions during the final two years of the grant period. 


Common Themes 

The diverse experiences of these colleges suggest that there is no common set of practices for 
achieving refonn. Nevertheless, some of the practices revolved around themes that were shared 
across all three institutions. The diversity provides an opportunity to explore how colleges with 
differing institutional contexts and foci approached some of these common themes. Two of 
these themes are discussed in the next section. First, all three colleges implemented Achieving 
the Dream reforms while focusing their efforts on a targeted group of students. Although their 
strategies and particular target groups differed in various ways, the colleges all maintained an 
emphasis on targeted implementation. Second, all of the colleges faced difficulties with leader- 
ship turnover during the Achieving the Dream grant period. Their diverse methods for address- 
ing this common challenge help illustrate different approaches to tackling an issue prevalent in 
community colleges. 

Narrow Focus on Student Subgroups 

While most of the Round 1 Achieving the Dream colleges implemented numerous strat- 
egies targeted at different student subgroups, each of the three schools in this analysis focused 
more narrowly on a particular subset of students. College A targeted those with multiple remedi- 
al needs, College B focused specifically on improving the outcomes of developmental math stu- 
dents, and College C focused on both developmental math and college-level math students. 

In tandem with their more focused aims, all three colleges in this analysis also coordinat- 
ed multiple refonn efforts around their chosen student subgroup. Many of the Round 1 colleges 
faced challenges in scaling more intensive strategies (that is, those that reached students for long- 
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er amounts of time). 2 All three of the institutions in this analysis, however, ranked among the 
schools that reached higher proportions of students with a more intensive strategy. Each had im- 
plemented a set of strategies at medium scale (reaching 10 percent to 25 percent of the target 
group) or a relatively high scale (reaching more than 25 percent of the target population). In addi- 
tion, their strategies tended to have a similar focus, which centered on providing more intensive 
academic support strategies, such as one-on-one guidance, to students in their targeted classes. 
These colleges also implemented at least one other strategy aimed at improving the success of 
these student groups. Among these were a summer bridge program, preparation for placement 
tests, technology-aided instructional supports, success courses, 3 and learning communities. 

While Colleges B and C focused their efforts on instructional supports like tutoring and 
supplemental instruction, College A placed more emphasis on student services. Despite these 
differences, all three remained highly focused on a targeted group of students early in the initia- 
tive. Their strategies for expanding these targeted programs to more student groups are also out- 
lined below. 

Emphasis on Instructional Support 

Colleges B and C focused specifically on math students and implemented a few key 
strategies aimed at improving students’ progress at multiple places in the developmental se- 
quence. College C identified a target group of students at both ends of the gatekeeper threshold 
in math (the highest level of developmental math and the first college-level math course). Math 
was selected because both data analysis and student focus groups identified it as a major barrier 
to success. As part of Achieving the Dream, College C decided to implement supplemental in- 
struction (SI) to increase the amount of time spent on math tasks and to provide student support 
connected to the classroom. 4 In the beginning, the college implemented SI in a small number of 
sections of the highest-level developmental math course and required a math lab for some stu- 
dents in the first college-level course. In each successive semester, the school expanded its SI 
program and required SI for increasing numbers of students. Math sections with SI also incor- 
porated a flexible curriculum of success components, including an orientation provided by 
counseling staff, in order to provide additional classroom support. 

This college also adopted several other reforms based on the target population of math 
students, including curriculum revision to incorporate software, a new placement assessment 
test aligned with the curriculum, and a pilot summer “boot camp” to prepare students in multi- 
ple levels of developmental math for college-level work. These additional efforts further rein- 
forced the college’s attempt to provide extra instructional support to math students. 


2 Rutschow et al. (2011). 

’Success courses are aimed at introducing students to college life and enhancing their study skills. 

4 SI provides support through additional review sessions outside of the lecture period. 
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College B initially focused broadly in the area of math before deciding to align multiple 
interrelated strategies around developmental math students; these students received additional 
academic supports from college entrance through successful completion of the developmental 
math sequence. These supports included (1) in-class tutors to assist students with assignments 
and to answer questions during lecture; (2) mathematics software integrated into developmental 
math courses; (3) case managers to monitor students’ attendance and academic perfonnance, 
and to provide additional in-class and out-of-class tutoring and support for students; and (4) a 
weeklong “Fast Track” workshop to help students whose math placement scores were close to 
the college level to bypass developmental math. 

Enhanced Student Services 

Unlike Colleges B and C, College A focused on the most underprepared developmental 
education students. College A changed its practice to allow early registration for such students 
and developed learning communities to target them. Many of the learning communities linked a 
developmental education course with a student success course. In the initiative’s early years, 
“success coaches” — faculty or staff tasked with providing case management to the students — 
were embedded in the learning communities. College A later transitioned from the case man- 
agement approach to a holistic advising model and hired eight full-time “success specialists” to 
mentor the students. All of these strategies aimed to surround the targeted students with multiple 
service supports connected to their academic activities. 

Broader Target Groups 

Over the course of Achieving the Dream, the three colleges gradually began to extend 
their targeted work into other areas of the curriculum. College A revised its strategy midway 
through the grant so that slightly more prepared developmental education students could be 
served. College B began to incorporate many of the developmental math strategies, such as tu- 
toring, into college-level courses. Similarly, College C hoped to extend and scale up SI to all 
developmental education courses (including reading and writing), if financially feasible. In 
these ways, the targeted Achieving the Dream focus of colleges became the starting point for 
broader influence across the institution. 

Management of Implementation Leadership Turnover 

All three colleges profiled in this chapter faced difficulties with turnover in leadership, 
including at the presidential level and in the higher-level leadership team (for example, vice 
presidents of instruction and of student services). Additionally, the leadership of Achieving the 
Dream changed multiple times at each of these three institutions: each school had at least three 
different Achieving the Dream coordinators during the first five years of implementation. While 
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each of the colleges confronted significant changes in high-level administration, they were able 
to manage these obstacles by designating a set of leaders and by developing a cadre of faculty 
that could help oversee various pieces of the implementation process. The work of the colleges 
represents different approaches to dealing with this issue. At the same time, all of the approach- 
es typically included similar elements. Among these were encouraging more faculty and staff to 
step up into leadership roles, providing them with incentives and support to do so, and organiz- 
ing their larger institutional accreditation efforts to leverage multiple initiatives in order to sup- 
port their student success agendas. Each approach is described in further detail below. 

Expanded Faculty Engagement 

All three colleges incorporated broad engagement structures to include more stakehold- 
ers in their Achieving the Dream work, though their specific approaches varied. College A 
serves as a particularly strong example of faculty coordination, as it established faculty leaders 
at each campus to manage the implementation of interventions and to oversee coordination 
across campuses. College B boosted engagement by compensating faculty to provide additional 
tutoring and by hiring new staff to provide other services. College C facilitated faculty engage- 
ment by developing faculty leadership councils that made specific recommendations to the pres- 
ident’s cabinet. 

College A had three different presidents and multiple vice presidents during its first five 
years in the initiative. The institution established a developmental education task force made up 
of faculty leaders across the college’s different campuses. These leaders helped facilitate and 
sustain college-wide conversations about the goals and practices of developmental education. 
They were charged with hosting a series of conversations for faculty across campuses about the 
types of practices that might help improve the success of students. As a result, the college estab- 
lished a unified voice about potential changes to programs and policies relatively early in the 
initiative, despite the fact that high-level leadership changed multiple times. 

Some administrators at College A were able to make an important impact on faculty 
and staff despite their short tenure. For instance, an earlier president promoted greater collabora- 
tion by restructuring his cabinet to include students, nonadministrative staff, and a faculty union 
representative. The establishment of a college-wide strategic plan also facilitated expanded in- 
volvement: broad-based strategy sessions for internal and external stakeholders at all three cam- 
puses helped inform goals and approaches. Another president later formed focus groups com- 
posed of faculty, students, and staff. In addition, monthly cabinet meetings were made public 
and often attracted an audience of 40 to 50 attendees who gathered to hear faculty and staff 
leaders present reports. Such practices helped build and maintain the voices of faculty and staff 
in the larger institution’s management and implementation of refomrs. 
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Promotion of Direct, Inclusive Leadership 

Unlike College A, Colleges B and C had greater challenges establishing this type of 
faculty-centered leadership during the early years of Achieving the Dream. These difficulties 
may have resulted from the colleges’ decision to focus on the involvement of high-level admin- 
istrative leaders, particularly because of their excessive turnover rate. In addition, institutional 
characteristics, such as location and resource constraints, may have made it more difficult to 
recruit and sustain faculty leaders in the face of such turnover. Regardless of the cause, both 
Colleges B and C had implemented few new reforms after their initial years in the initiative. 

However, both Colleges B and C made substantial implementation progress when a 
higher-level leader (either a president or a vice president) provided a vision for the work, as oc- 
curred during the last two to three years of the initiative. Within both institutions, these individ- 
uals retained enough authority to take charge of reform efforts and to provide direction. At Col- 
lege B, progress in implementation occurred when an associate vice president, who offered a 
pronounced vision, was put in charge of the implementation of Achieving the Dream. At Col- 
lege C, the president was the key figure who instigated a turn toward data-driven decision mak- 
ing. He began to reorganize teams into councils that worked on specific items and that made 
recommendations to his cabinet. Neither school, however, made much progress toward imple- 
menting reforms until this leader was established at the college and it was clear that his tenure 
would be stable. 

Once this leadership had been established, Colleges B and C moved to engage a set of 
middle-level managers (for example, directors and department chairs) along with faculty and 
staff to help implement specific interventions. For instance, at College C, the president was very 
intentional about gathering infonnation and feedback from the faculty, staff, and students. The 
president and his cabinet drew on issues raised in community forums to inform the development 
of a strategic plan that did not previously exist. Faculty expressed particular appreciation for 
having ownership of this process. 

College B used monetary incentives to engage faculty and staff in reform efforts. A 
number of faculty members were paid to spend an additional hour a week in the student success 
center, where they tutored students, or in the advising center, where they helped place students 
into the correct math courses. In addition, the college hired a number of part-time class tutors and 
case managers to support the implementation of their advising and supplemental instruction in- 
terventions. Further, they also brought on part-time data analysts to track students’ use of these 
services. Without the monetary support provided by the Achieving the Dream grant, the college 
believed that it could not have motivated staff to do the extra work or have hired additional staff. 
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Targeted Faculty Development 

College A instituted its faculty developmental education task force around the same 
time as Achieving the Dream began. Faculty representatives were charged with setting annual 
goals and collaborating on strategies on a college-wide basis. Some of their activities included 
working on adjunct faculty handbooks and adjusting cutoff scores that determined which stu- 
dents would have to take developmental math courses. College B supplemented direct academic 
support strategies with targeted professional development for faculty and staff involved in those 
strategies. This brought together all math faculty each semester, including tutors and managers, 
and eventually evolved into a training day focused on strategies that were determined to have 
shown promise or success. Through these types of indirect efforts, the colleges worked to build 
capacity, engagement, and continuity in order to deliver services to their targeted students. 

Building on Accreditation 

Accreditation played an important role in supporting broader-based institutional support 
for reform at College C. This institution coordinated the accreditation process with Achieving 
the Dream and also used the process to foster strong early buy-in, to motivate faculty to focus 
on the success of math students, and to become a more data-driven institution. College C hired a 
data analyst and began to measure course-based learning outcomes in its targeted math courses, 
in addition to student outcomes like retention and persistence. 

As part of the accreditation process, the college also began to use a formal development 
model, required for new strategies or policy changes that called for concrete, measurable out- 
comes and evaluation activities. The faculty in each department became responsible for the pro- 
cess and the data. Each department was also responsible for developing outcomes specific to its 
goals but connected to the overall mission of the college. The model emphasized the use of mul- 
tiple measures to gauge success beyond students’ final grades or course completion. The formal 
model also emphasized measuring student learning outcomes within individual courses. The 
faculty team identified learning outcomes for each course and used department-wide exams to 
test the outcomes. 

The math faculty’s enthusiasm for this process inspired other departments to try it, and 
by the end of the study period faculty members had established a set of core competencies for 
each of the general education departments. Each set learning outcomes for their courses that link 
to the general education competencies and developed common departmental exams for core 
courses that test these learning outcomes. Full-time faculty also gathered periodically to review 
student outcomes on departmental exams and to discuss how to strengthen their efforts for the 
given competency. 
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Key Lessons 

The different approaches adopted by the institutions analyzed in this chapter present some over- 
arching lessons for other community colleges. First, the colleges’ ability to manage the reform 
process during periods of leadership turnover suggests that leadership can be effectively lever- 
aged at multiple levels. Committed, engaged leadership among senior administrators can drive 
successful initiatives, but turnover at this level is also a reality for many community colleges. The 
colleges described in this chapter worked within their unique contexts and supported multiple 
individuals who showed leadership capacity, regardless of those individuals’ official positions. 

Additionally, the colleges made a concerted effort to focus on important student sub- 
groups. Each college also began to expand the reach of these practices to larger groups of stu- 
dents or faculty after gaining experience with the initial subgroups. Targeted professional de- 
velopment for faculty and staff involved in the initiative supported this focus. In one case, the 
college also leveraged the accreditation process to coordinate its reform efforts. 

The three colleges highlighted in this chapter had diverse experiences and operated in 
disparate institutional contexts while participating in Achieving the Dream. Their stories exem- 
plify how different institutions can work toward improvement in a variety of ways. Despite con- 
siderable challenges, community colleges are thinking creatively about their leadership models 
and are building support for targeted reforms. 
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Chapter 4 

Conclusion 


In many respects, Achieving the Dream can be thought of as a continuous improvement model 
for institutional reform. Colleges that use this model commit to establishing a culture of evi- 
dence, to engaging faculty and staff broadly in developing strategies that support student suc- 
cess and institutional reform, and to adopting an iterative process of program improvement 
using data. As documented in an earlier report on Achieving the Dream, Turning the Tide, 1 
many of the 26 Round 1 colleges demonstrated progress in building a culture of evidence. In 
particular, college leaders committed to investing time, personnel, and resources to endorse the 
core tenets of the initiative, and they focused on a student success-oriented agenda. The colleg- 
es strengthened their capacity for institutional research, which improved their ability to collect 
and analyze student data and to use these data more frequently to inform policies and practices. 
Most colleges also implemented a variety of strategies designed to improve student success. 

In other ways, however, Round 1 colleges did not fully realize the vision of Achieving 
the Dream. Many of these colleges, for example, were still working toward firmly establishing 
important facets of the model at the end of the implementation grant period. In particular, strat- 
egies to encourage students’ success generally had not yet reached large proportions of stu- 
dents. A number of colleges also struggled to engage faculty in using and applying data and 
research, despite their commitment to this key tenet of Achieving the Dream. For instance, 
surveys of faculty and administrators suggest that faculty tended to use data on student pro- 
gression and completion much less frequently than did administrators. 2 Similarly, though col- 
leges generally improved their institutional research capacity, they were not expected to — and 
generally did not — rigorously evaluate new interventions to better understand the effective- 
ness of reform efforts. 

In short, translating institutional reform into practices that substantially change the ex- 
periences of large groups of students — and observing changes in institution-wide student out- 
comes — may simply be more difficult, and may take more time, than first anticipated. This 
realization, however, is not unique to Achieving the Dream. A number of related interventions 
in kindergarten through twelfth grade (K-12) schools have promoted the use of data as a strate- 
gy to improve schools. Research on these interventions has found that (1) the underlying theo- 
ries that translate data use into school improvement are under-conceptualized, (2) the evaluation 
designs for these efforts are generally weak, and (3) there is little evidence that student out- 


Rutschow (2011). 

2 Jenkins and Kerrigan (2013). 
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comes improve following these interventions. 3 Increasing the capacity of schools and colleges 
to gather and analyze data may be a necessary condition for improvement, but it does not appear 
to be sufficient. More work is needed to understand how such reforms might translate into insti- 
tution-wide gains in student outcomes. 

Achieving the Dream encouraged colleges to focus their strategies on subgroups of 
students, faculty, and staff — for example, revised orientations for first-year students or re- 
forms in developmental math — and to scale up the strategies following indications of effec- 
tiveness. This process takes time and, given the targeted focus, improvements resulting from 
Achieving the Dream efforts may have occurred on a smaller scale than the institution-wide 
analyses of student outcomes in this report can detect. Several other important changes also 
occurred during this period that might have affected students and exerted pressure on commu- 
nity colleges, making it more difficult to isolate the effect of Achieving the Dream. Notably, 
the Great Recession started toward the end of 2007, and during the course of the study the col- 
leges experienced large population shifts. Rising enrollments were accompanied by state fund- 
ing cuts and shifting demographics, with the proportion of minority students gradually increas- 
ing over time. These and other changes may have exerted pressure on the colleges that could 
have affected student outcomes. 


Lessons for the Field 

Despite these competing challenges, individual colleges made important strides in improving 
institution- wide student outcomes, as illustrated by the experiences of the three colleges profiled 
in Chapter 3. In many respects, the lessons that emerge from these experiences correspond with 
— and also build upon — conclusions from Turning the Tide, which made recommendations 
covering four broad areas: 

• Improve the leadership model, including broadening the model to include 
faculty, staff, and lower-level administrators in order to focus more directly 
on improving instruction and student services, and providing more support to 
colleges that are experiencing leadership turnover. 

• Bolster the data analysis model, including developing stronger supports for 
colleges with weak institutional research capacity and incorporating more 
nuanced measures of student outcomes that relate directly to classroom prac- 
tices and students’ learning and progress. 


3 Cobum and Turner (2012); Turner and Coburn (2012); Marsh (2012). 
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• Advance the broader engagement model, including concentrating efforts 
to involve adjunct faculty and staff and to provide in-depth and meaningful 
professional development to all college personnel. 

• Enhance policy, including developing a more systematic approach to evalu- 
ating community colleges’ strategies and interventions. 

This report extends these recommendations and illustrates promising approaches de- 
signed and implemented by the Round 1 Achieving the Dream colleges. These approaches are 
now considered important practices in the field, as highlighted by the stories of the colleges pro- 
filed in Chapter 3. 

Aligning Reform Efforts for Systematic Change 

It may not be enough for colleges to implement a variety of programs and services if 
they are not well aligned. Research in K-12 education emphasizes the importance of imple- 
menting instructional programs in a coherent, cohesive, and aligned way, so that reforms work 
toward the same goals and interact with one another to be mutually reinforcing. 4 Supplying 
targeted students with necessary supports may require such coordination. Further, the integra- 
tion of multiple initiatives, policies, and practices may require a more general redesign of pro- 
grams and support services. The literature on organizational effectiveness and improvement 
suggests that no one practice or even set of practices may be sufficient to improve the perfor- 
mance of an operation as complicated as educating students in a wide variety of program fields 
and types. 5 Improving performance may then require more fundamental systemic refonn — 
reviewing and redesigning practices, policies, and processes over time and realigning them 
with new organizational goals. 6 

While there were some instances in which Achieving the Dream colleges brought about 
systemic changes to programs and services, most of this work focused on the early stages of 
students’ college experiences and did not address challenges students might face in later courses 
or in staying on track toward graduation. For example, a number of colleges redesigned their 
student intake process (for example, orientation, placement testing, initial advising) or enhanced 
instruction in developmental education or gateway college-level math and English courses (for 
instance, providing supplemental instruction and introducing learning communities). However, 
few colleges focused on curricular and instructional reforms in college-level courses beyond 
math and English, though some did under the aegis of other initiatives such as accreditation or 
other grant programs. This is significant because, like developmental college students, the rna- 

4 Newmann, Smith, Allensworth, and Bryk (2001); Boreman, Hewes, Overman, and Brown (2003); Bryk 
etal. (2010). 

5 Bryk et al. (2010); Jenkins (201 1). 

6 Jenkins (2011); Kuh et al. (2005). 
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jority of students who are assessed as “college ready” when they enter college do not end up 
completing credentials. Moreover, many such students, particularly those in transfer programs, 
report being confused by the array of choices available to them and lack strong guidance from 
their institution on how to negotiate college successfully. 7 

A number of Round 1 schools also found it difficult to engage large groups of faculty in 
the reform agenda. This limited engagement may be partly due to colleges’ emphasis on early 
stages of students’ experiences, which may be relevant to only a smaller proportion of faculty, 
such as those teaching developmental education courses or introductory math or English cours- 
es. Similarly, the broad institutional performance measures in Achieving the Dream, such as 
persistence and completion, may not resonate with faculty who focus on issues related to cur- 
riculum and instruction in their own disciplines. 

Building on Accreditation Processes 

In contrast, one college used the accreditation process to engage faculty by focusing on 
measures of student success and by using data to inform improvements in classroom practice. 
This college’s model also emphasized the measurement of learning outcomes and eventually 
spread beyond the initial group of faculty into other departments. Learning more about how 
colleges can link and integrate a focus on student learning as well as on student progression 
and completion may help to identify ways of engaging faculty as central actors in future re- 
form efforts. Those types of experiences may also illuminate strategies to incorporate research 
and data — as well as to implement interventions that are data-infomied — more fully into the 
reform process. 

Engaging Leaders Throughout Community Colleges 

This report also highlights colleges’ efforts to broaden their leadership framework to in- 
clude faculty leaders and middle managers in the reform process. Effective leadership can occur 
at multiple levels within a college. 8 Presidents and high-level administrators play a key role in 
institutional reform, but given the decentralized organization of many colleges, deans and de- 
partment chairs can also be critically important for mobilizing faculty and staff. Cultivating 
leaders among the faculty and student services staff, however, may also be an essential step in 
institutional change. Such leaders can sometimes spur change more effectively among their 
peers and within their departments. Colleges that cultivate strong leaders throughout their insti- 
tutions may have a greater chance for long-term success, even when some leaders leave and 
must be replaced by others. 


7 Grubb (2006); Gardenhire-Crooks, Collado, and Ray (2006); Scott-Clayton (2011); Karp (2013). 

8 Kezar (201 1); Grubb (2013). 
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Starting with Key Student Subgroups 

While improvements in institution-wide student outcomes likely require changes that 
affect large proportions of students, different students may be better served by different reform 
efforts. The three colleges analyzed in greater detail in Chapter 3 demonstrated varying ap- 
proaches to reform, and they often tailored new practices to the specific needs of targeted stu- 
dents. They also focused multiple strategies on high-need subgroups of students — primarily 
those placed or enrolled in developmental courses — and consequently reached those smaller 
groups of students by means of diverse and coordinated services and practices, such as academ- 
ic assistance provided through advising or supplemental supports offered in the classroom. 

Integrating Research with Reform 

In general, there is a continued need for research on community colleges that is well 
aligned with reform efforts, in order to understand whether reforms are working. Developing 
future refonns in tandem with rigorous research designs would help fill this need. Although in- 
stitution-wide student performance is unlikely to change substantially if large numbers of stu- 
dents do not experience something different, even large-scale refonns will not produce change 
if new innovations are ineffective. Furthennore, ongoing, careful, and rigorous evaluations need 
to be prioritized in order for college leaders, faculty, and staff to leam whether reforms are ef- 
fective. Knowledge about program effectiveness and common drivers for organizational change 
is growing, but there is still much to leam, especially with respect to institutional change in or- 
ganizations as complex as community colleges. Rigorous research focused on the impacts of 
specific interventions that directly target student experiences can inform colleges and the field 
about the effectiveness of these new efforts. Such research can provide colleges with specific 
and accurate feedback that they can use to infonn decisions about programs and services. Shar- 
ing this knowledge could help other colleges implement and scale up more effective practices. 
When refonns more directly target institutional change and only reach students secondarily — 
such as those that promote increased institutional research capacity and data use — changes in 
student experiences may occur less directly. In such cases, research on the process itself and on 
measures of institutional change may be more useful. 


Looking Ahead 

Achieving the Dream was founded upon a belief in evidence-based decision making that has 
resonated with colleges across the country and that persists today. The initiative has helped 
community colleges establish a solid foundation for reform. It has helped colleges to enhance 
their institutional research and data capacities, to develop stronger cultures of evidence, and to 
pursue a variety of new refonns. This is an important development in that many faculty and 
staff are increasingly focused on data and its implications for student success. Achieving the 
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Dream also serves as an important learning network that can be used to share lessons about re- 
form efforts. Through this network, colleges can build on successful efforts that have been un- 
dertaken elsewhere, and they can also leam from past mistakes made at other colleges. 

Achieving the Dream has grown considerably since its founding in 2004. The initia- 
tive now stands as an independent organization, stretching to nearly 200 colleges, 15 state 
policy teams, and more than 100 coaches and advisers who participate across 32 states and in 
Washington, DC. The 26 Round 1 colleges were leaders in this effort and learned alongside 
the Achieving the Dream partners about institutional change in community colleges. The 
Round 1 colleges’ innovations and their willingness to publicly share their experiences have 
pushed the field forward and helped advance reform efforts in community colleges. Other 
system reform initiatives, such as the Developmental Education Initiative and Completion by 
Design, have been heavily influenced by lessons from the early experiences of Round 1 col- 
leges. Indeed, Achieving the Dream colleges continue to be leaders in new programs now un- 
der way in the field, including in initiatives that, like Completion by Design, emphasize fo- 
cused pathways for students. Among these programs are the Statway™ and Quantway™ 
Networked Improvement Communities and the New Mathways Project, 9 all focused on re- 
thinking traditional approaches to mathematics. Colleges, policymakers, funders, and re- 
searchers across the United States are building on lessons from Achieving the Dream colleges 
as they think about community college reform. 

This report marks the end of a long-term study of the first colleges to join Achieving the 
Dream. The initiative was designed to tackle persistently low rates of student success in com- 
munity colleges. The Round 1 colleges have demonstrated that, though change can occur at the 
institutional level, substantially improving institution-wide student outcomes is much harder 
than was envisioned at the start of the initiative. 

There is still much work to be done to improve the prospects of students who enroll in 
community colleges in the United States, but Achieving the Dream has established a solid 
foundation and a strong network to embark on that work. 


9 Statway™ and Quantway™, created by the Carnegie Foundation for the Advancement of Teaching, are 
one-year courses that fulfill college -level developmental math requirements by focusing on statistics and quali- 
tative reasoning, respectively. The New Mathways Project, developed by the Charles A. Dana Center at the 
University of Texas at Austin, is built around three accelerated mathematical pathways and a supporting stu- 
dent success course. 
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Appendix 

Additional Analysis 




As indicated in Chapters 2 and 3, researchers conducted analyses to investigate factors that are 
potentially associated with changes in student outcomes at Round 1 colleges. Using field visit 
implementation data and survey data, this analysis examined the core activities related to the 
Achieving the Dream initiative, whether the influence of certain activities was associated with 
student outcomes, and whether other findings emerged from the reanalysis of the field data. 

Theoretical and empirical research literature was reviewed to establish hypotheses 
about institutional characteristics and external influences relevant to community colleges that 
may serve as key factors for refonn. First, factors associated with institutional progress and 
student success were identified, and then qualitative data collected on Round 1 colleges were 
coded into measures reflecting these key factors. The following factors were operationally 
defined, coded, and analyzed: 

• Leadership capacity — Evidence that an institution has strong, active admin- 
istrators who communicate clear goals and plans for improvement and who 
support faculty and staff leaders to help them carry out these goals and plans 

• Institutional alignment — Evidence that an institution’s reforms are planned 
and implemented around well-aligned common goals across different initia- 
tives and processes 

• Data capacity — Evidence of an institution’s ability to manage, analyze, and 
report on data 

• Academic support systems — Evidence of an institution’s ability to offer co- 
herent academic programs and services aimed at improving learning and stu- 
dent success 

Evidence of the above hypothesized factors was investigated through the reanalysis of 
data sources available from college field visits and data submissions, including: 

• Qualitative field data collected by MDRC from 2006 and 2009 at Round 1 
colleges 

• An nual reports submitted by colleges to Achieving the Dream from 2006 to 
2008 

• Results from a survey of faculty and administrators at Achieving the Dream 
colleges 1 


'For more information on the methodology of this survey, see Appendix A of Rutschow et al. (2011). 
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A list of the measures that were part of this analysis follows. Analysis of these factors 
using the five main outcomes yielded no clear relationships between implementation and 
changes in student outcome. 


Faculty/Administrator Survey Scale 

A selection of survey items from the faculty and administrator survey, listed below, were used 
as part of a correlation analysis to determine whether any survey items relevant to the factors 
listed previously were associated with student outcomes. Confirmatory factor analysis was 
employed in order to reduce the number of items that measured common variables. All survey 
items were standardized before being summed into scales. 

• Data structures — whether data in the college’s student information system 
is accurate; whether the college’s institutional research department is ade- 
quately staffed and is responsive to requests for infonnation; whether data 
and reports are user-lfiendly, clear, and provided in a timely manner 

• Data procedures — use of data in decision making; professional develop- 
ment on data and research; perceived usefulness of, and buy-in to, data and 
research. (These subfactors were also analyzed individually and are detailed 
further below.) 

o Use of data in decision making — whether the college evaluates the ef- 
fectiveness of its educational programs and practices; whether the re- 
spondents’ departments use data and research on students to make deci- 
sions about curriculum, teaching practices, academic support, program 
planning, academic program review or evaluation, advising, and/or iden- 
tification of students who are struggling academically 

o Professional development on data and research — involvement in train- 
ing or professional development related to institutional research, data 
analysis, program evaluation, or assessment 

o Perceived usefulness of, and buy-in to, data and research — whether re- 
spondents believe that research provided by the college is helpful; 
whether they believe that asking faculty to participate in regular discus- 
sions about data on student outcomes is useful; whether respondents do 
not use data or research because they feel it is outside of their job respon- 
sibilities 
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• Organized discussions — frequency of participation in organized discussions 
held at the college on the improvement of academic achievement or closing 
achievement gaps, on the academic needs and performance of students of 
color, and on the academic needs and perfomiance of low-income students 


Implementation Data 

The following items are based on an analysis of qualitative field visit data from each college, as 
well as from annual reports submitted by colleges to Achieving the Dream. 

• Presidential stability — lack of president/chancellor turnover between 2004 
and 2009 

• Accreditation alignment — aligmnent of Achieving the Dream goals and ef- 
forts with accreditation goals, processes, or requirements 

• Initiative alignment — alignment of Achieving the Dream goals and efforts 
with other institutional reform initiative(s) 

• Collaboration — increase in collaborative efforts among divisions or de- 
partments 

• Integrated institutional research — presence or development of an institu- 
tional research department that is integrated with other departments, such as 
information technology, institutional effectiveness, or planning and assess- 
ment 

• Intervention strategy target — implementation of an intervention strategy 
that is targeted to developmental education or first-year students 

• Curricular strategy — implementation and scaling of an intervention strate- 
gy that focuses on changes to classroom curriculum or instruction 

• Student support services strategy — implementation and scaling of an inter- 
vention strategy that provides student support services 

• Required orientation — presence or development of a policy that requires 
that all students, or an identified group of students, participate in orientation 

• Required student success course — presence or development of a policy that 
requires that all students, or an identified group of students, enroll in a stu- 
dent success course 
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Implementation Codes 

Following implementation field visits in 2009, the researchers who conducted the visits devised 
a series of quantitative codes for each college. Each college was coded on a five-point scale in 
47 areas, indicating level of implementation of Achieving the Dream principles and influence of 
Achieving the Dream. Two researchers coded each college and then cross-checked codes for 
reliability. Researchers also looked across colleges to ensure codes were applied consistently. 

• Leadership commitment — level of support for a focus on improving student 
success from the president, from the senior leadership, and from faculty lead- 
ers (for example, deparhnent/di vision heads) 

• Institutional alignment — level of collaboration between student services and 
faculty/instructional staff; extent to which the college has a strategic planning 
process that uses data to inform measures and goals for student success; ex- 
tent to which external grants and the college’s accreditation process support 
efforts to improve student success; extent to which meetings, organizational 
units, and work groups regularly focus on student success 

• Data structures — ability to meet the demands for data and institutional re- 
search; integrity of policies and practices used to collect and securely store 
data 

• Data procedures — extent to which faculty and staff receive training on us- 
ing data and research; extent to which college evaluates its programs and 
services to see if they improve student success; extent to which college uses 
data on student outcomes to develop or refine policy and strategies 

• Professional development — extent to which professional development train- 
ing reinforces efforts to improve student success 


Key Intervention Strategies 

The following items are based on an identification and analysis of colleges’ three primary 
strategies (referred to as “key strategies”). Prior to the 2009 field visits, Achieving the Dream 
team leaders at the colleges were asked to identify and rank the top three strategies — those in 
which their institutions had invested the most time and resources. These strategies were selected 
from a list of all strategies undertaken during their five-year tenure with Achieving the Dream. 

• Number of key strategies that focused on changes to classroom curriculum or 

instruction 
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• Number of key strategies that provided student support services (including 
instructional supports) 

• Number of key strategies that reached students for 10 or more hours per se- 
mester (“high intensity” strategies) 

• Number of key strategies that reached over 25 percent of their intended target 
populations (“large scale” strategies) 

• Presence of at least one key direct program and services strategy that was of 
high intensity and large scale 

• Presence of at least one key direct program and services strategy that was de- 
signed to last longer than a single semester 


Variation in Pre-Post Change Between Colleges 

The following two-level model was run to estimate the significance of variance for the change 
in outcome means from pre-period to post-period. Level 1 is for the student cohorts, and Level 2 
is for the individual colleges or sites: 

Level 1: Y Lj = £j =1 ccj SITEj + T \j + s Lj T Lj + t Lj (l - 7y) 

Level 2: /3 j = y 10 + Vj 

Yij is the average outcome for cohort i in site j 

T L j is the postintervention time indicator for cohort i in site j 

(3j is the average pre-post change for site j 

SITEj is an indicator for site j 

t tr N(Q, 4>jl ) and s i j~N(0, &f / ) are two separate Level 1 residuals, for those in the 
preintervention cohorts and postintervention cohorts, respectively 

Vj~N( 0 , rf ) is a Level 2 residual 

The estimated cross-college variance in pre-post change scores (f 2 ) was found to be statistically 
significant for all five main outcomes in this report. In other words, there is evidence that 
t 2 ^ 0; that is, the pre-post changes in outcomes varied across the colleges in the study. 
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About MDRC 


MDRC is a nonprofit, nonpartisan social and education policy research organization dedicated 
to learning what works to improve the well-being of low-income people. Through its research 
and the active communication of its findings, MDRC seeks to enhance the effectiveness of so- 
cial and education policies and programs. 

Founded in 1974 and located in New York City and Oakland, California, MDRC is best known 
for mounting rigorous, large-scale, real-world tests of new and existing policies and programs. 
Its projects are a mix of demonstrations (field tests of promising new program approaches) and 
evaluations of ongoing government and community initiatives. MDRC’s staff bring an unusual 
combination of research and organizational experience to their work, providing expertise on the 
latest in qualitative and quantitative methods and on program design, development, implementa- 
tion, and management. MDRC seeks to leam not just whether a program is effective but also 
how and why the program’s effects occur. In addition, it tries to place each project’s findings in 
the broader context of related research — in order to build knowledge about what works across 
the social and education policy fields. MDRC’s findings, lessons, and best practices are proac- 
tively shared with a broad audience in the policy and practitioner community as well as with the 
general public and the media. 

Over the years, MDRC has brought its unique approach to an ever-growing range of policy are- 
as and target populations. Once known primarily for evaluations of state welfare-to-work pro- 
grams, today MDRC is also studying public school refonns, employment programs for ex- 
offenders and people with disabilities, and programs to help low-income students succeed in 
college. MDRC’s projects are organized into five areas: 

• Promoting Family Well-Being and Children’s Development 

• Improving Public Education 

• Raising Academic Achievement and Persistence in College 

• Supporting Low-Wage Workers and Communities 

• Overcoming Barriers to Employment 

Working in almost every state, all of the nation’s largest cities, and Canada and the United 
Kingdom, MDRC conducts its projects in partnership with national, state, and local govern- 
ments, public school systems, community organizations, and numerous private philanthropies. 




